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Introducdcnt Where Do We Go From Here? 



David J* Bafie^ 

In Jime (rf* 19%, tte S€C(»idCm»limi C(Hif(»€^ 
f^d at the Univeraity of British Columbia, tte first tmving bee^i m the 
UnivQcsityof VictcmamI989« Many erf tte papers fhmitt^fimocmfeiin^ 
publi^^ in a specif is^ ctf the Alberta Journal cf Educational 
Research. Tliis preseitt doctimrat is a ccanpil^ion of of tte ^&pm whidi 
were pres^ted at the second conf^^ce. 

The ccmfetence was held at a time when educatic»ial n^asmmient, aiKi 
classnxMii testing in i^Btkular, was undogoing a ladtc^ change in both th^»y 
atd jsBctke. The iiuroducticm of the ccmcept of ''authentic^ measnranont in 
Beyond Standar^zed Testmg (Archbald & Newmann, 1988) has provkied the 
stimulus fcr an emsmoi^ MfX in tte a>i^m:t of ecfaK^^kml measurement as it 
is pfac^iced in dassrotmis* This shift has been {femcmstiBted in ev^thing from 
the cm tent (rf the annual meeting {Ht^ram of the Natioi^ CcHinci I fcnr Afeasuie- 
ment in EdiK:atl(Ki to the documents surrounding the Year 2(XX} initiatives in 
British Columbia. 

In onkr to fircititate discussion, particsi^ts in this confeiem^e w^ sent 
copies of Year 2000: A Curriculum ondAssessrmm Framework for the Future 
(Ministry (rfEducatic^, 1^9a) and Tte Pr/moo'^'w*'^ (Ministry (rf Educa- 
tion, i989b), the draft pnqxx^d for the annmim:^ changes to the fnimary 
program in British Columbia, as a stimulus for confereiK^ papers. These 
documents include recommoidmicms am! even d^ands that teachers woik 
toward "authentic** assessment i^iocedunes (1985te, p. 17). They include quota- 
tions sm:^ as *^as$essment and evaluation fechnkiues must mirror the actual 
learning experiences of the child..." (1989b, p.l3), "fw mc«t assessment 
purpo^ in school, ir^ticmal fcmns of standardized te^ aie not very useful, 
and they are inapprt^riatc in the Primary Ptx^ram...** (1989a, p. 17), **thCTc is a 
shift from examinMion to demonstration...'* (1989b, pJ3), **comi»risOT with 
other students, or the assignment and use of letter-grade or pseudo letter-gmde 
symbols is alw irwppn^iate for primary childrCT...^ (1989b, p.l3) and 
''although checklists are useful for teachers as a way of organizing information, 
they are not aR)ropriate as reporting devices..,** (1989b, p. 166). 

The stalemCTits from the Year 2000 initiatives are representative of much of 
the discussion regarding classroom-based educaticHml nK^uremcnt in Canada 
at the {Hesent time* Many of tl^se evali»iticMi issues which are prcrfiferating the 
present educational scene att£^k the roots of tn^doi^ measurement ttecHy and 
pr^lice. For at least the last two decades* most measuremCTt specialists teve 
concentrated tt^ir efforts on large-scale, standardized* selectton-type assess- 
ment techniques; large-scale, "high-stakes** testing programs; and methods that 



omipitfei' tedmoiogt^ tmve availa^ This gitiBHion is imcteaamMble 
sim^e ti^se tl^ areas t;^^^ fam&ig aiid 

fioom pdit&uans and go vemme^ 

d^litya^imbltepen:qttkmof(fediniiigst^^ Hcm^vw, as edt^ators, 
particmlariy d^srocm teacl^s, 

disg^mtted with wfa^ tl^ pm)etvc to be a waiped picture ol ^mfents ami 
edQc^n in generd wfakh 1ms r^iitted 60m tl^ exchmve use the type 
iiifcmDaticm |m)vkied by tl^se laige-scate, ^amimxiizs^ s^ection-type tests, 
tte need fm a radical ^ift in thinking and piBctice has ^^ged. 

Measuiement specially in Canada* latho: than opposing many of the ideas 
of the '"attfhratic** movement thM sXtBck j^t practice, imve rapidly cmfinned 
ami ^Kfcvsed the f^ic {rfiitosc^y of ttese ideas. However, thew same 
^^eciali^ are wcricing to en^ne tf^ new {»acttce$ utilize uliat is already an 
extendve tfi^ory md kmmtedge i»ise of edu^ 

ignorii^ an that has gone befwe and starting ai^w. It is essratia! that a 
desmictive levoluticm in ediK^atioiml n^^urement be avoided. What is neces- 
sary is a ratioRal, but fairly capkt evcrfutim. Within this context, the Second 
Cmadian Confi^i^nce CHt ClasatXMn Testing was (XHivef^, and it is hcq>ed that 
this collection of pa|)ers can ccmtribute to the present evolution in measurement 
ttecKy fflui practke. 



Archbatd, DA. & Newmarmr RM« (1988). Beyond Standimiked testing: Asie$m\g 
a^fhcr^acadsmkacluevemefUm Reston,VA:Nactona! Association 

of Secondaiy School Principals, 

Minbtry of Education. (1989a). Year 2000: A cumctiSum and asiessment framewcnk 
for the future, Victoria: Author* 

Minbtxy of Education. (1989b). The Prirmary Program. Victoria: Author. 



Dr, David J* BatesoHf the organizer and chairpmcm for ihc conference, is an 
Associate nt)fessor in tl^ I^artment of Mathematics and Science Education at the 
Univdsity of British Columbia and a Research Associate with both the Educational 
MeasuiememResearchGroup(EMRG)and thcCcntrefor Applied Studies m 
(CASE). 
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THE CONTEXT OFCLASS ROOM PROCEDURES IN 
EVALUATING STUDENTS 



Robert J. Wilson 
Queen's UnimsiQr 

MiK:h of the altsitkm dev(^ to classroom-bas^ assei^oit 
C(Htcentrat» cm tlie tempter: wfmttheteacherdo^.whattheteachermightn^, 
and what tits teaci^r shouki 60. 

Varicnis aigge^kMis have beai made to teaAos. In recait currkulum 
(focim^ts te^^hos aie advised to use a wide range of assessment tools, 
concoitrating on obsrarvfO^ and prcx^ measures, and to ^(»e the lecx^ 
individual accomplishments in stuttent (tevetopment files (B.C. Ministry of 
Educ^icHi, 1^). Tacts for courses in educational n^asiBement generally 
feature adaj^tiims of i»)«hometrics smi {Htrfe^ifmal pr%;tfce fo clasnooms 
(Brown, 1981; Cunningham, 1986). Odier writers, convinced th^ the acbi^ve 
mettod has not adapted well enough, have used a omii^rative analysis oi 
classroran practice and measurement standards to reccmimend where and Iww 
a IWW mdding of theory and practice might profitably occur (Frisbfc and 
Friedman, 1987). 

Two a^umptiOTis about classnxMn-ba«d assessments are usually mmle in 
tl«se efforts. First, the clas^txNn teact^ is pafranming the evaluation activity 
prumrUy to infcmn judgements about stucfent progress m that instruction and 
learning can proceed more effectively. Secondly, die teacher is virtually a free 
agent in determim'ng which jmrticular cyaluatiOT activities and puqx^ will be 
adqpted in the classroom. 

The purpose of the present paper is to investigate these two assumptions by 
referring to woric we have done and are {nesently doing that is aimed at explwing 
tiK! brofKler environm«it in which student asKsstnent occurs. ' 

What do Teachers Want to Do! 

That teachers wish to fulfill the first a^mption about the value of evalua- 
tion has n(^ been validated by much research. We are presently examining the 
attitudes of 101 Bachelor of Education students during their year of teacher 
prei^ration. As part of their partfcipation in this study, these piwspective 
teachers were individually interviewed halfway through this year with an 
instrument designed to explore the reasms for the altitudes they hold toward 
evaUiating students. (They also completed a Lik«t-typc alutude scale on three 
separate occasions during the year.) 



• This work, supported by the Social Scieaices and Mathematics Research Comicil, 
includes among its members, Ruth Rees. Marilyn Cannock, Lyn Shulha, Alison Taylor, 
besides the author. 



purpo^ for aifmmistmi^ evab^icm mstnimeitts to dd^iea. Tto fiiufeni 
tea^CT* le^ts (ti^isfi^B^ iidoiBiiksfin^ 

m iHOvi(^ 'm TBiAc I di^)^ by level d faileiifkd {HEc^ice. Sftatettis 
pcq»riiig to teach from KiiK&^Egaitefi to Gisdb 6 me labdbd PJ (J?simsy- 
Jfu^). Ttee» pnqimng to iem:h tmn Gmdte 4 tfuxNig^ 10 aie Uielled JI 
^imkir-lQfemKdiate), and tiKise inqiaring to teadi fmn Grate 9 through 
Ontario Ac^tenk: Comsc (fbm^riy Grate 13) m:c labdled IS (Intem^di^ 
Senter)« 

Table 1 

A COTiparison Among Student Teachew Conc^nlng Importance of Various 
Re3»om for Evaluating Students. 



(I^ Giv^n in Ranked Otter) 

RcasonsT ?Vi JPs IS's 



To check students* progress against 2 1 l(tie) 
cmir^ ob^^xtives 

To compare students' achievement 10 10 10 
toothers 

To generate mari« for reporting 9 9 9 

purposes 

To ensure that students (k> assigned 7 8 7 
w<xk 

To prepare students for this kind of 8 7 8 
evaluaticM! in the fuUiie 

To have ^iKlents practice or apply 6 4 5 
what has been learned 

To diagnose students' weaknesses 3(tie) 3 3 
with tl^ material 

To enable stuitenis to monitor their 5 2 l(tie) 
own {progress 

To help me decicte what to tc^h next 1 6 6 

To allow me to see how welU taught 3(tie) 5 4 
the material 



the i^ii^i^ ii!qx)i«m:e of varies piipo^ cS wBimkm Bt 
level within tte sy^on ttey inteml to iw»k* Key difioem^es se^ to exfe* 
b^:ween the PI ^iK^ttsaiid oth^ two gnni|» cm tl» ^ 
'"to he^ 0% decide what to teadi c^** and "lo 

own pn^ness,'' but on most of the si«^sted jmrposes. Utile disi^ieenm 

The highest nmkcd piinx>ses for alt three groups can he int^iM^ as a 
goioti tendency to see evaluatioa activity as infcKmhig tte teaching-leaniii^ 
poce^ Pispwes with this ov^all fimcticHi in miirf (c*^>raHnrfCTWce» 
applying learning, diagnosis, self-nKsiitoring) are mme highly wei^ited than 
tlK»e who»fuoctkH» is mcmadmini^tive in nioure 
{o€ future exams, mrnn-refereiK^e)* 

In this reflect, tt^se stiK^t teacNn agree with their cdieagues on 
ciKTiculum-wriUng temis, evaluatkKi text authcHS, and instructors in measure- 
ment ccnirses coK^CTiing tte nm^ purpose (rf evaluaUcm in cl^^ to 
infcHtn tte te^tng-leaming (nrocess. 

^Mlat Do Teachers Actuatty Do? 

The group they do not agree with » well we their more exp«icnced 
cdle^ues. Virtually the same item was us^ in ajHOvious istudy (Wilson, 1990) 
in which 51 jH^tising teacfim in Bdti^ Cohunbia and Ontario fttled in the 
same scale (presented as a cl^k-li^ of ptnposes) fcM* each instmment they 
administered to their closes during a repenting p^od. The 24 te£K:hers 
ccmprising tte Ontario sub-sample were divided imo two groui^ of 8 clen^n- 
tary (Kindergarten through Gra(fc?<^airi 16 secondary (Grade 9 through Oracfo 
13)te£K;her$. Table 2 shows ihccomi^daMisoflherwikctf frequency of puip^ 
of the piBCtising te£^hers with q( the relevant student tether comparison 
groups. 

CcHnpared to all the student teeK:hers, the practicing te^m rate ^e 
gcneraUon of marks fw reporting purposes'* much high^. Indeed, this purpose 
domiimt^ the entire exm:ise fcr tlie expertenced teach^, it having been 
checked as a purpose on four out of every five instrumrats. Otb^ disparities 
between the practictng teachers and the stuiknt teachers grow laigeras the level 
of intended pr^tice tr^rea^s. 11^ diffenuices are most pn^unced at the 
seccHidary tevel wheie iImj highest ranked purposes for evaluaticm given by the 
practicing te^h^ rcfar to administrative and external aims, all of which are the 
lowest ranked purposes for the prospective tesK^hers. 

Our study of the^ 101 te^ha^ is continuing. We will interview as many 
of them as become employed in the province of Ontario twice more during the 
coming school year. In this way we hopo to determine how they ad^ their 
present views on assessment to the reality of their own classroom Ufe and also 
try to determine why their views alter if they (to. 



TaUeZ 

A Cmiparison erf Plractklr^ Teadim (PTs) and Stitdmc TcsKilwre (STTs) 
Com^ii^ Importam^a of Vmicnts Reasons fer Evaluating Scudents. 

(D^ Given in Ranked Chder) 

FTs STs 
ELEM SEC PI JI IS 



To check students* 4 2 2 1 l(tie) 

{TOgPcss against 
course d:^tives 

To compare sUMtenls* 10 6 10 10 10 

achievement to othoB 

To generate marks for 3 19 9 9 

lepofting purpo^s 

To mmx^ that stiHtents 9 4 7 8 7 

do assigned 

To piep^ students for 8 5 8 7 8 
this kim! of evaluatiOTi 
in the future 

To have students 2 3 6 4 5 

pimrtice w ^ly what 
has be«i learned 

To diagnose students* 1 8 3(tic) 3 3 

weaknesses with the 

mat^iai 

To enable students to 5 7 5 2 l(tie) 

mctfiitor their own prepress 

To help me decide what 7 10 \ 6 6 
to teach next 

To altow me to see how 6 9 3(tie) 5 4 
well I taught the material 
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What Acajitnts far the Differtnces? 

Thnra^ (»her «^ we ore ck»iig (Re^ 19S9; WUscffl and Rees, 
expect thmttelr views mH alter. We have hypothesized tha; the police and 
IHxxxdiH^s concerning audcni achievemenl devolved upon teach«^ from 
levds '^ve'* than is iIks af&ninimaUve bieruchy will force their evaluadon 
activities into tebuivdy narrow areas. 

Table 3 

Content of Policies and Procedures iti Sttdent Achievement by Levef of Or^. 



Ccmtem 


Nfinisdy 


Levrf 
Dfatrid Schod 




CJassnxnn 


Schdarshtps 


X 


— — — ~ ■ 






Consultation le 
Evaluation 


X 








Appeals 


X 








Reporting, Grading 


X 


X X 


X 


X 


IndivtdDat Assessment 


X 


X X 


X 


X 


Examinations 


X 


X X 


X 


X 


Piroifioiion 


Y 


Y 
A 






Attendance 




X 


X 


X 


Communication to 
Students 




X 


X 


X 


Weig hting of 
Eva) Liadon Types 




X 


X 


X 


Timing and Types of 
Evaluation 






X 


X 



Tat)le 3 shows the policy content and the level at which this content becomes 
part of the student aswssment environment for teachers in two districts and 
several elementary and secondary schools in British Columbia. (Our analysis 



erf tbe (^^^ ^ivttYmm^ is yi^ ccmii^ed) Iti^ is a my heavy 
OBptiasis in the policy and jmK^hird firam^vofk m examiiwicms, iqxHting 
fc»rmats, imxnotion* fiml evahi^km welgfals. Mi^ ctf (he content (rf these 
pdk^ cieates a stMKtodgmkm of ^pros^ (ccmimm examin^kH^. se4: 
tmtdn^ for repwtmg, and nwid^ xepoiting symtx^) ttm fot^xs the 
as^ssn^ ac^v^^ (rf teaches c»tto id^veiy i)^^ 
Our aady of tte poik^ aiea (tf ^itt^ achfe wii^ 
a nusn^ch aimmg (a) few poUcy ^^em^tts thtt (Hovitte frfiilosc^cal 
dkecUcm tote^riters; (b) tlK^actii^ 

iMly beyo^ the[»i]iiffly grmles) mu^ inq>Ien^t; and (c}cv^! ediK^iOicmai 
gc»tst8lra^ittsft[»'iniUfcediK»ticmint}i^IH^ 1^}. 

Forexampte, during the 1987/B8 sdux)l year, tte B.C Nfinistiy ofEducttion 
h^apolk:ywhid)advoc^d^f(dlawingg^: ^'Itise^^uialthmimientsbe 
kqft infimud of i»ogie^ ^teved toward exp^:taUc^$ hekf in comnum by the 
teac^t stiKiait, parentSt mid ctmunimity,*' In the actual implemratatkm of 
r^)orting to i^iiQQts, towe^, all schools in ^ 
level im>vidcd lelmive jud^ementSt usually hi lett^ gn^ fcmnat, tb^ defined 
expecttfkms in pen?mtage range equivatents. (In fact, m:h a ddinition is 
ccHnptetely dicu^ as rather letter grades nw percoitages are cfeflmd in terms 
other Itan each cHher.) 

The ^rectficity and standgrdizatiOT of mosi of the procedures at the school 
Icvd, ami the linking of the^ pixx:edurcs to ^)ecific calo!^ 
attention to a school-year rhythm based m the leporting cycle. It might be 
assunoed, tt^, that teachm could fulfil the rqxHting lequiiement with instru- 
ments thtf were cmy to prq^ie and mark but which may or may not be related 
to the ^tual learning going on in the classroom, (Muticulm^ly if theie weie m 
cater dear expectatkms fcH* the evaluative process. What is more serious, 
howev^» might be the pc^t^lity that a low expectatim level for the evaluaticm 
gradually comes to replace a mcHe ambitious expectaticm for the i^tual learning* 

Other investigators (Carter, 1984; SUggins, Griswold, & Wikelund, 1989) 
have stK)wn that the fevel (tf as^^n^t typically employed by te^hm at all 
levels is quite low in tmns of (X>gnitive (^m|^tty« In cm data, it is clear that 
the cognitive demand of ^ngle-wcmi c(Hnpletio^ and shcHt-answer items 
(quc^tknis which ^:q>caied(Hi 44% of the instruments we c(rflected) is notlikely 
to be high if for no other reasi^ than that the format does not allow any high^ 
level than lecall of specific bits. 

The same case fc»r multiple-choice questions* and for other suf^ly and 
pedcHrmance items, is not so easily macfe. Here, for example, is a multiple- 
choice item lakra from a Gr^e 12 examination: 

1 . What is the distance between the points (1 , -8) and (-4, 4)? 
A. 5 
7 

C 13 
D. 17 

It may be that the student faced with this item selects a formula from 
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n^moiy, ^t^ku^ the rdevant Ma from the ^mi, and sekcts the arawer 
ch^e^iohidc^ralmkni* Itm^^bettecase^lKiwev^.th&fta^iMii^witf^ 
Ui^ fmrnub* bat with a ioKwi^lge erf tte Pythas^raem theornn, q^es thai 
kmmtec^e to the $»n^ reqt^ oihI ies|NHNb m exac^y the aBne way ^ i»r 
feUow^iKient lt]^:»K;h^(mtobem^isthttUisdifTk»Utoc^^ 
que^ioiis lAiased hi shismm^oer aaytbtogmlk^ 
used by sUKfents to tespmid to the item. 

Two ccHKiIti^H^ sem fair fcH* the types evaluatim inmm^ts we foumi 
in our earli^ wc»iL Fmt, many items allow studeitts Uxtemmstiate c^ly a tow 
levd cognitive cq^ei^kms. Seccmd, ttose thm do allow fmrmoa^ omii^ 
q)^tic»^ may iKApnyvute imamMgiKMis data afxHit tli^ (^leittkms. 

White it wcKik! scan (^ious diat leccmim^idmions to 
encoirage better int^i^c^Mimi wcniU be welccmie (tte wwfcof Biggs and C6liis 
(1982) is especiaUy mxtewcHthy hctc), m:h recomn^mteti(»is asanui^ tbm the 
IHiS5entmo(^isnotnmii^te»:her5*needs« But ifttete^:b^* real nt^ds arc 
to iiK^ rqxHthig (femamb for stngie lid>el jtidgranents of stiidmts* lelative 
sttttding, md to accmiplish that task with a minimum ctf tinK^ qpent cm it, then 
lecommatdmiaas for mc»e involved develoinnent, sccning, and inteqMetation 
of complex items will seem, at best, initating and inelevant. Pt^h^q^s the le^on 
why tl^ cognitive level exhibited in classroom assessment instnmtents does not 
alter miK:h through the schxi cycle is that such growth is neither rcquirod tm 
expected by the evaluation poKcy of the sctoo!. 

Canchmon 

ITiese results, admittedly fragmentary and jwneliminary, may nevertheless 
{Hovide a cautionary note: Befcne tbo^ of us tnteiested In the evalumion 
activities of teachers in classrooms fnoceed too far down a road toward 
implen^tation of new^ af^ro^tes, vfc might first attend to the broiler 
envircmment in whk:h these ^vities occur. 

It may be that te^^^ use the results of classroom assessments (ot reasons 
other than those posited by documents, experts, and naive i^ctitiontt^. Unless 
tlK!^ cnvironnKi^nt of classrooms can be altered so timt certain administrative 
functions concerning reporting, attendance, and communication with outsiders 
axe less overwhelmingly intrusive, it will be non-productive to work with 
teachers aloi^ to change tteir present pr^tice. The te^hm, at least those we 
have wcHked with, are not free agents to make the types of changes outside 
deem d^irable. 

It was a major breakthrough to umiterstand that tl^ clas;»oom itself im a life 
tlmt sh£Q>es many of the ^tivities that go on th^, including evaluation of 
stutfcnt learning. Now it seem*? that a classroom can be seen as a unit in a teger 
structure which also creates a community, a commtmity with very intrusive 
expectations. Attending to that larger unit, and altering what it consich^ 
necessary and desirable, may well be a jmequistte to successful change in 
student assessment practices at the c]asmx>m level. 
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yessn. Victoria: Minfecry cf Bdiicarion- 

and Q>Ub, ICR (1982). EpakMhi the qualty feanung. New York: 
Acacfemic Press. 

&Dwn, RO. (1981). Mcostmng clmwom odm^^nenL New Yorfc: Holt, Rlt^hart 
and Winston. 

Cunnln^m, G.K. (19K)). E^MaOton and pgyciuA^^ mmsmement* New 
Yofk: N4^^nillan. 

Fryrfe, D^., & Frledmmi, S.J. (1987). Test standards; &Miie Implicatlcsw far the 
measuiement cuniculum. E(hicaticmdMeastiremau; hstm m^Pr^^ 17*23* 

Rees, R, (1989). The mfbdeiux of the prcndndd educmUmd system on educators* 
tifohmion pofides and practices. Paper presented at Ontario Educational Research 
^Xmncil Annual Conference^ Toronto. 

Sdgglns, R,J., Griswold, MMu and Wikelund, K.R. (1989). Measuring thinking 
skills through classroom assessment. Jotmud of EdMcotiofwf Measurementt 26{3), 233- 
246. 

Wilson* R.J. and Rees, R. (1990). Charactemtks <4 t^jUcj procedures got^mmg 
smknt eifak4ation. Paper presented at the Annual Meeting of the Canadian Society for 
the Study of Education, Victoria. 



ERIC 



14 

10 



Student Evaltiatk>n in the Ungraded Primary School: 
The SCRP Principle 



Lcs McLean 
Ontario Institute Studio in EducaticHi 



Ungr^kd sduxils follow nmiurally from other devek^mients that teve been 
sh^^piimary^lKX^ in roc^t years. CtmtiniKHis pragma has always bem 
a fact of schoc^ life« but teldy it has been givm oflkml status md saiK^tic^. The 
id^ of cunkulum keyed to schod yeais is inconsi^t with tli^ ap^^ 
teacbkig of leading, writing, listening ami i^^eaking km^wn as wMc kmgm^ 
pedagogy^ and all sit within a still evdving ccmcq^t called activity based 
leanUng. ImpleniemtiK^withapoItcythat^udentswithhand^ 
kiiKls will be tmigbt in legularcla^srooms ^ mtK^h as possible (main^ieaming), 
aiKisctooIing c^ganized byscfKxrf yearmim finally be ^^oKkmed* 

With the departmie of gr^ levels^ many tradstkmal ways oS defining ai^ 
evaluating stm^t ^^hievement go as well. Hie ok^ d)vi<His is the gcade 
ecpiivaloitscOTe. Gr^equi\^entsinredding have beencriticizedby language 
tlK^msts ami cuniculum spcckili^ (a a ctocade, so their demise is to be 
weteomed. Hieir shaky statistical pmpertk^ ami poor substantive latic^e 
make them $uq)ect in every object Ama^im^lemi$thmtheirexistem:eami 
use for so many years has created tte belief that iK:hievaneiu in the primaiy 
^tKX>I can be ciqHured by numbers accurate to tm or thiee signiflcwt figuies. 
**Jm^^s reeling level is 5.4, ^kI she is only in Gr^e Hueet** A similar belief in 
IHecise numbers has led to the creaticMi of other test scoria <ising the so-called* 
"item response the<My^ Wtoi the Americans came up with the iwne, **Rit** for 
their scale unit, far westm Can^ was re^y with its ''Brit'*. AI^« school 
acfatevemfnt is miK^h more complex than that — and C(»isiderably less precis. 

Gmde equivalents will persi^ (with Brits and ottor fictic^s) as tools to bri^g 
Ofder out of ch^ until we can offer teachers and officials something good and 
pn^tical to lepiace tt^. The purpose of this is to suggest a principle and 
some impltcatic^s for good and pr^tical rq[)I^ements. 

The SCRP Princ(f>le 

Measar»n^t specmlists have IcMig advocated toting as the main tool for 
student evaluatkm, and te^t^ do consUiK^t and administer many tests. In the 
primary sdiooU however, tests have i^v^ been teadi^* evaluaticHi method ctf 
ctoice. Obs^ation and informal evaluaticm of student work listed as the 
dominant methods (WahlstrcMn, 1977, Fair et al., 1980), At their best, these 
metlKxfs provicfe authentic as^sment of Khievement« Unf(xtunately, they ate 
not always at their best, neither cl^ical nor modem test theory has had 
anything to offer in suggesting impiovem^ts. The challenges of informal 
e\^luatton aie not best (tescribsd in terms of reliability and validity; better 
ccMicepts are fairness and thoroughness. It is very diflscult to objerve and 



15 



ev^^ iidimfi^y in an mive iffiirary 
j^mfems are treats evgiilmiKtecUy 

Tte c^ptexity ami cfiflftnilty dT i^imary cbssrocmis rote mit ^iiq)ie« 
uniform pro^tniBS my kind, iiK^iMiuig evalaaticm m^hods. No om 
iq^nx^h wiOwcfffcinaUcIassancxnns. In siK:h a ^umtra»ii^ust»lly turn to an 
cngmkii^ prindi^ that can giik!e deciskni m^dng white jNi&^rving rtexibil- 
ity. Te^^fmwtom^tiKni^^ofdmoesperc&y umk^ 
coifralted ccHidiikms^ WkiHelmgmge^^imgislms^mmAufmmi^ 
with some ctetaitediHtx^ediiieswOT^om 

m tte $p(H« The guUing (»inci{rie is tf^ language is mrt divisiUe, Uutt i$» 
re^Kimg, writing, lia^niing and ^peakhig aie hitemm^cted and (tepemtot in 
som kmwn and scmie stilt my^erkH^ ways. Swix a tHim:i|^ has {m)found 
ini[^catkms for pedaigf^, requiring ra^:^ly diff^^em te^hing ami testing 
ni^thocbthanwwtt^ccms&imisclKm^CHily adecacteago(McL^ 1988). 

Whote langu£^ te^hing and te^ng en^Aasizes the communtealion q{ 
mewing in cmtexL Assessnentta^mimth0refcHefHesentac(Hitext,middng 
ti^ longer and mote complex. StiKl^ts tmng difTermt bfK^grtmds to the 
taskSt fflhl this natural variati(Hi rediKres trEKliticHia! relialnlity mdtees (such as 
iittOTialccMiststeiK^y). In ^ccmd language testing^itseemstbattl^bett^thetest 
(the niOTe it reflects communicative lantgii^ the^) the lower will be the value 
ofCnNri3^h'sA^(Swain,19^)* Assesmient thai faithfully reflects stiutent 
p^ormam^e in meaningful contexts is refeimi to as ''auth^tte% a quality 
c^HK^ ck^ly related to validity but better suited to ^htevement teeing 
(Archbald & Newmann, 1988). Most test (tevelqpni^t efTcHt in Ncmh America 
and Eon^ is now going into tasks r^uiring students to (tenKHii^te abilities 
aiKl ddlls directly, in other words, by perfonmmce. Directly int^jnetable 
evii^ce is i»oving to be mcne mcful and aur^tive to educatois than indirect 
and abstr^t test scores* 

Activity based learning introduces aiK>ther aspect that shouki be mentioned 
befwe considering assessn^mt prim^iples. In c^r fcr activity based teiBning 
to succeed, the te^ea* has to est^l tsh an auno^riw^re of reiq)ect and Uiist in the 
classrocm. Students havetoagreetoli^entoedcbotho^^topraiseandtocritic 
with re^)ect« One {wticulariy successful teacher spoKis much of tl^ first two 
wedcs d'the school year building such an atmc^;rfi^ by setting out some rutes, 
gettmg ^udents to i^opose othe^ and by starting i^tivities that illustrate the 
need fw such a ^irit (McLean, Aitken, Van Duzer and PeterscMi, 1^). In the 
beginning, the tc^icber will assign sUidents to woik together on a common task 
in small gn}U[». 11)^^notthefamiliarredbirds,blt^t»rdsandotherreadin 
groiq>s^ which are a form of at^tity groiq>ing. Each grtmp ^ould reflect the 
ability range in tin; classroom as cloi^ly as pc^ibte. Socni, students are workirig 
in diflt^ent groupings as tad^ change, or w(xking alone on an assignment. 
Classnx^n atmost^i^ is a tope in it^lf , but here we have to move cm with this — 
brief menticm. 

The asse^ment principle suggested to go with these changes is that evalu- 
ation should be based on a SysiemaHc Cumulative Record of Performance^ 



SCRP, E^ctfthewcmlsbasbeenc^fiiltycfKM^rainli^ 
mi goi^al princi{^ erf* fabn^ mid tiionntgimess, so will be disc o ssed 
intioiL Tti6di«:iise»mib»iU30tli^ifwet^t^^ Ttesetting 
f<^thediscu^(mis8^iii^tobepriiiwysct^ — the Tim three or fraryms 
(^schodii^ a£tm age 6. 

Record of Perfdrmance 

EvahmtiGH] can imi sIk^ be on i^timl s^ptes of i^mlCTts* ^mxk — 
(SI iteirpedcmnaiK^d'ineaningfid tasks. Tewhminiteeariyyeafsdomudi 
ofth&iK)w, with i»i»:kni5 litttegukbacctowitmi^tbestri^tt^ 
dbne* Six-year-olds diaw picmues and dictitte tte n^mii% to thdr tefK:ha' m 
cMh^aduItt fc^ examine, iml the adult writes tt^^^sttHy" cm tl^IHCtuie (crcm 
theback). WcmlsslKmkl always be a^ociated with a slosy^aiHlS^^ 
valued as mich m iOoiks read theorid argues that stodes tokl am 
^^)eeially goodi^nn^^ a pen^r^ food fcs tte miiKlt thm stcmes aie the 
iHiilding bkxdcs of a comi^te elonentary ^:bool cumculum. Rk* evalu^km* 
"'one might pmiculaily focus m ... ^miething written, chamatized, diawn, that 
gives evictence the effective effea (tf tt^ toiit white drawing c^i suf^xHting 
knowtec^, skills, and so cmi** (Egan, 19S8, p. 247). 

Pi»fi»m»u:e on meaningful tasks i^iould becaptmed in portfoIk)t$^ recwds 
of ^hbvment r^tabi^ and managed by e^h Wfent* Note that it may be a 
recc»d d*p^(Hmance th^ is retain^ by the stu£tent« ratter than the peifcmn- 
ance itself. Wten s(nnething is '*dran^ized^ f(»r examine, the p^cnmance 
maynotberetainedatalL Victeocamefasar&becomingmcHeandmosecommai 
in elemoi tary schools, so e\^ tte dramatizati cms cm be c^mued, but the viiteo 
cassette need not be kept in tte pcHtfcriio. Tte exi^^ice oi the cassette, am! 
perhaps its location, is recmded in ttepc^oUo, becai se tlm pcKtfcriio is ttebasis 
for both fcmnative and ^imnMUive evaluatimi. Students will have two das^ 
ofp(»tf(rfi(^ — i^tive and cumulative* Cuniratpn>^cts,wc]dcstnim)giess,are 
k^ In the active pc^oUo, and sample to be retained in tte record are 
trBUisfored to the cumul^ve poatfcdio when re^y to mov^ on to ffliother topb* 
Wten wmkmg in a gnmp, ^:h ittucfent makes a copy of his perfcnmance 
for iiK:lusk)n in tteir recmi. P^ormam^ is often a group efl(^ but individual 
accountability is ^sociated with b^r learning (Slavin, 1987). 

Such gr^ reliance chi pcHtfc^os dictates that th^ be some b£K±up. What 
if a stiKient loses ter or his cumulative pcmfoUo near tte (md of a tcmi or school 
year? As di^us^ in (fetal! below, evaluation of poitfotios is a periodic pint 
effort of teffi:ber and student At evaluation time, the teiK^her should make ami 
retain summary notes, the main purpose of whk:h is backup in ci^ the actual 
ncotd is lost The teachCT's notes would be a poor q^ximaticHi, but with the 
student's help, they coukl be used to reconstruct a reasonabfe summaiy of 
achievement up to the time of latest review. A teacher's note might lo<* 
^HTiething like this, the suident having seen (or heard) and agreed to it 

RoA reviewed with on ^ (date) . Tte recwd was up-to-date in math 

and science. Quality was uneven, with a few excellent and a majcnity good 
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r^ew. Rmlingli^n^t inited^biUvmtuigranil^weiieatoKMiaclda^ 
Mait A I»€smt was agreed to be m^giittl lo good 

dooc in the United Kingdom, stssiaig mm tban 10 years ^ in Scotland. 
Rocof^ became edi^aticmal pcdicy a fe w 

axKt natkmal testing have dirown dmdH m their fitture. A ccmiBidi^ve 

evahiitfiOTyascmiedomintocalairtNykfesmax^En^ 

iqxsrt of that evaluati<m gives bc^ history ami {Hactical infimnaticm ^xmt 

ittiengths and we^kx^is^ (fi^tmd&xAt 1988)* All may mA be km, tKnii^ver» 

because evidence jim appeared thM leccnds have wider s^Ircation than firtf 

tl^mght* The dqiartment of edocaiitm at tl^ Univ^ty oi Camls^s^ has 

intiodiiced leconbctf mAievemont fco-abrat 60of t^ 

in Educ^icHi stm^ ts as part of tteir jHi^rvlce training. same benefits are 

seen fcH^ gr^iates {Hiqsaring to ent^ teachii^ as Ux d^mentary aiKl 

seccHKtey ^Kxd stiM^ts. 

We h^ thm these {Rofttes will hdp stiKtoits to set goals f^ 
see what they have learned ami to ic^tify the areas in which they can (fovelq> 
fintha-* We also expect \bm by ccmtpiUng a prt^U, a stuctent will get an idea 
of wh^ it is like for chibbm to ccmipiete recoards of ^hievements* It will also 
give stncknts a flavcmr of what ^^misal may entail when it eventually 
arrives (Bearc&Hi & Rem, 

In the US., 50-75 c^leges ml imivmit^ are using them to evaluate the 
testing pedc»in^K:e of jMofessors (Watkins, 1990). 

What we see emetging is that evaluatkm can Hnally become an inte 
of te^^lung ^ learning, and that evaluaiim, teaching and learning can all be 
usefully ccmcdved within theoretical fnmieii^^E^ based on language i» mean- 
ing. Thec»i^kaIfTafi%«mksfH)tcmlyteUu^ 

alsoknvtostnictureandsctHethetffiks. Teaching nuiyyetgetbeyomlcrBft and 
bec(Hne a i»rlesskm (McLean, 1985). 

Cumubtive Records 

Research in cognitive sdeiK^e has illuminated imny aspects of meaningful 
learning, and one a^iect tl^t stands cmt is the emphasis on tt^ cumulative nature 
ofteaniing. Newmatcmlmustbeintegnited with what we already know if it 
is to be remember^ ami ^lied. Imieed, since the need to r^te to pri(Mr 
learning is so strrag, it is a paradox that leming gets started at all Itie whole 
ccmcqrt meaning is that we make sense of iK)vel infc^mation by relating it to 
whatweakeatfykiKw. Howdoesleaminggetstarted? Itisespecially clear that 
Imiguage c<»npetence is an outcome of a slow cuumulative process, much of 
which is still mysterious. 



* This work, «ipported by the Sodal Sciences widHu^ 
mcludes among its mmbers Ruth Rees, MarSyn Connock, Lyn ShuUw. and Alison 
Taylor besides the author. 
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provide an mMbemic a^^^^i^ measne. evra if the t^s incfivkb^Iy are 
aidl^it^ Much C3f the leamif^ti^fi^ t^im {dace wUl be stt^ 
MiK^tts kkntify and expiidUy tecord tte links mmrng ibek pedomm^^ 
smq>tes. lliis amy be a case v^^th^ measiiietR^ 
sxmA fi^ Ufiks may Uknt^ c<mn€ctians which would otherwise go immHiced. 
I f so, llmk mmwh^ b^to:, sifK:e is the (^.^ctive, not scnne scit of 

i<teal measuremem« Genitive sctenti^ have also ^iqrfma^ the c^idi^ 
reftec^tion cm play in learning, Stcqfq^faig ami *^kx^ ig badk**, ttmt is, Ihinkii^ 
t^cl^isminqpwtant^q>in(%»mdkli^^ BuiMingtl^ciinmla** 
tive pcmfoUociBi givef^imary sthocA {Hiplse^y pr^tic^ {»^:tice in reflection 
and taxcsKmy-buildnig. 

Until stififents get used to keq»ng pcxtfoltc^ teacters will have to qiend 
scKi^tini^explainingmtdillustiBtingttefr^. Ttemecbsiicsctftheactiveamf 
cumulative festers ane sinipte oiCHigh, hvt cmisicteraHe discusstcsi will be 
required ^xmt the oumulmivefcdcte, Thet^NreT^lainsthatiii^ wc»fc(»i 
a iqpk: project U finished (or ^0{H)ed, at teaft), a 

best wc»ic should be saved TTic wmk saved ^mikt show what has been 
ao^omfrfished, what soluticnis were ot^ained and what it means. PHmtuy 
teacb^ will i^ed a rq)^ire df^^ples to get tf^ proems started 

SUKtents illustrate tl^ metming of a p^ormmices^nple during a review by 
^wii% how tte samfte is liidced to w^ abeady in the pc»tfc£o, CH)vi<msIy, 
this evolves slowly fi^ six-yeai-okk. ScMmdmes wten wosk is started, 
links will be few ami difficult to ^k>w, and this work will be noted as needing 
nuseexpkinatkm later. Ttewcml'^noverc^beintr^dtK^edafterayearOTtv^^ 
inthesenseof''new;ofanewkind(»'nature;hith^unkiK>wn". Students team 
that whtt is novel to (MieperscHi may be wdl-kiK>wn tootle. Itfoltowsthat 
every new sample ^kted to a p(»tfotio is an occasion forreftection. 

The ImportatKe of Being Systematic 

Sy^ems are r^uired to cfeal with the comfriexity of primary classrooms, ami 
tcacl^ are good at figuring cnit systems. Th^ have their reccmi bodes, 
mHebooks, folders and the like. What few teachers Imve is a sch^ule fcs* 
evalumic^ and a system of receding that c^ tmmc even mid thorou^ 
coverage of their clashes). Tem^hers already have to assume mcne of a 
managerial role in ^dvity learning, mi they quickly learn that when 
managers have too miK^h to do, they deflate. In this case, teachers delegate the 
firstleveIofrec^keq)ingtotbestmk3it$byme^$ofp(MtfoIios. Twotypes 
of systems are nee<ted fx evaluation, (a) a way of describing contoit expecta- 
ticms, and (b) a schedule. We assume fi^'tl^mimcsit that studoits are abcacfy 
familiar with the pufp(^£ and operation of portfolios, and that the te^hci' has 
been at^e to estatdish fji atnK)^h^ of reflect and coqieraticHi. 

Content expectations. Thee^enc^ofimgr^ted schools is that studrats^art 
at different stages and {mceed at dt ffer^t rates; there are no fixed expectaticHis 
by school year. TTie teacher mustNiveexpcctations, of course, and ideally these 
should be taiI(Hi^ to each student (^>^at]onally, this is a nightmare, and it 



becomes smsc m m^sats dtv^ge in ihsk pe^fioiiMnw ovct several years, 

is wh^ pcmfolios ami tl^ SCRP isfa^^ 
Bxpe^^atkms ^ 8^ m t»n^ (tf cofflul^tve gra^ 

Tq»J^ ommmtfc^ expe<a^icms to ptq^ and i^t^ 
(tfttetype^ofperftnimicess»ni^t^ Enwiples 

come to sdKx^, mimba a&ds^ sckmx ^pcftowes tho like* Tb& 
tsakxi^ k dom bet^i^eii teacher and mxOm m review ti^s, ^thast sfsecifKS 
tms«tsaiesetfOTthetkiieGftheiieiaievte^^ Sy^0ntffa:,mnttl^veiRDgEess 
isd»goal,af^^iKteittsieiKiilyuiicte'Samlit^ UiKkmmdiiigttto 
IH^rtice comes with tin^ mid the qTpcvttmtty to cbss^ nmmom 
examines. The dmllenge (nothmg ii^) is fc»^ tte 
j^KKtetis* 

Staiidffids ^ set in tetms (tf stming pc^ts and beitchmari^ They are 
OHnmimicAEed by means of nmaiive ctescri{Hions aiKi mnj^ pwtfoUos. 
Teacfim mtmbe^ loccmstiltacoUection (^p^mfcrfim in tte^^hooIcH^ district 
o^Rcc. CoUcctii^ will incitH^dx and nif^n^thpcKtf^ 
mariung {miods) for waiting poinu mA m these: 

- Suiikmts wto ccMild re^ wh^ th^ crane to sctod, had English (c^ 
FieiK^h , that is , the langu^ of in^nicticHi <^ the school) as their fust Iangi»ge 
and oiontfitcred no major oteiacl^ ^justing to scbocrf. Included would be 
examfrf^ofslow^avt^e and rapid im)giess» These are tc^ benchmarks. Call 
them TI,T2 and T3. 

- Students did ncA read wf^ they can^ to school but who were 
otherwise tike the fln^ group. Scnne will equal the top benchmarks in quality ctf 
pCTfwmance at the end of the period. Call these Ml, M2 and M3.tte middle 
bechmarks. 

- StiHic^ts whc^ Hrst language is not &igli^ ch* wtK> have other 
piKrf^s getting started in schawl. Call tbe^ Ol, 02 and 03, the *'other'' 
benctmiarks. 

The B(»id of Echic^ion fcs^ the City of TcMonto has just publidied a 
ccmpiebensive s^ies of beiK^hnmrks for language ami matt^atics perform- 
az^e tasks after three and six years of »:tKKrfing. The range of perfc^mance is 
huge^ as you wcnild expectt and ^w what haRwns cvct in "gnKied" schools. 
If they had portfolios, alt three of the grou{» would be well represented. They 
iUu^te how perfOTmance tasks can be used to suggest reascmabte taigets for 
stuitents, 

Re\4€w scheduk. It is easy to wcwk; out that if a portfolk) review lakes 10 
minuteSf and there are 24 stucknts in tt^ class, fcmr hours will be spent on each 
rouiKl (Previews. Ate^hf^inprimarycmcountonatinostflwhcnirsp^day 
oftimesuitabtefOTreviewing^only 15 tours in thiee weeks. Aqi^rt^ofcla^ 
time will be spent reviewing pcntfoli^^. If tl^ are 30 students, five hours will 
be required, accounting for one-tbiid of class time. This can cmly be ccmadmd 
sertously if the time is s^so seen as jmme instnsctbnal time, ai^ that is precisely 
tl^ claim that is made ((x iL The point here is that te^hm Imve to plan aiKi 
scl^ule reviewing at least carefully as they do my other activity. 
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Tte SCXP ]piiu^)te sUs in the inaimtr^n of citnm &;vek^mieiits io 
stiK^ evahMiUm - p(^(OTiiffliC6 te^g 11^ evidoK^ is ^liftfaig (fecmvely 
BWdy firan test scenes to directly inteqsi^sd^te evid^K^ in tte fcnm ctf peffcmTi- 
aiKe ofl meaningful tasks, Thisisn^ahu^sfaiftfis-prmiaysdh^ 
of coiHise, becmise they imve at ways (fep^tffed iffl ot^^ 
in tfi^ evali^toos. Wlmt is i^w is tte rK:<^itksi that evi<teiice torn 
perfcmnam^e testiiig can be <»ganiz^ 

tea sccHes served, Imtbett^. Tb^s^id^ithastobem^thiaatr^iitiofialtett 
cm also be meaningful am! that answois to the questkms mc a (om oi 
perfwnmm^e. Tei^ i^^eis ttmt have bcm n^ufced, p^fmi» ^im^ated^ are 
cimdklatesfOTiRchisioninieamis. Wcmiiecogmdcm^^^ellkig^antfm^tfc 
tests are not good candkUues, however. Ttey do not qualify as authentic 
measurement, and the time r^iuired to ccmpiete than is better spmt in (Mber 
ways. 

Stucfent evaluatk>n serves muiti{de purpose. Its pdmary piHpose is to 
siq^iy feedb^ to stiKb^ts^ teacter and i^rrats, but increasingly it has to 
provide ^ccHmtafnlity to the wid^ ccmimiaiity wto pay the lai^ cost 
educatkm. In onier to meet aU these ^mwds, there mt^ be a careful record of 
stiKfentperfOTmanceamia way toioffiqjretthatrec^. The pcqmlarity of te^ 
scores can be explaii^ in Imge by the ease with whidi they c»uld be 
reccHd^ and stcsed and by tli^ range of int^i^etmbn sd^n^ devised for tl^m 
by the te^ng {HofesstcHi. On!y gradually dk! it become ctear that tt^ craven* 
iefK:e and surface credibility of test $c<»es bn^ught at the price of 
authenticity. Rec(»rds of pc^ormance in the fonn of portfolios can provide 
auttentk documentation of achievement ami also i^vi(k summaries for 
puiposes of ^countability. 

Thekeytopectegogicalsucc^istoaotsurethat the portfolios are cumula- 
tive, in tte intellectual as well as the physical s^^. This means that stiKtents 
and teiN:f^rs wcnk tc^ther to give m^ng to Uk^ cumulative pcHlfoIio, asking 
wh^ efich new cntiy fits with the others airf gradually ccmstrwting a content 
nmp of the recc»d. Ccmsuucting the content map Imngs into play the powerful 
learning tool of reftecticm and ensures that the evaluation task is also a 
knowtedge prcxhictian task. The |Htx:ess can ami stould begin in the primary 
school, so that it becomes secmd nature to all students. Some of the mc^t 
important tecdiing mid l^uning will h^^n as students review their portfolk)s 
with the teacf^ and with othar sttKlents^ 

Rk:h cifinulative portfolios canncH micfge in large cla^s unless attended 
to systematically by te;Echer and students. Tt^ tether has to establish an 
atmosi^re of trust and sharing in which students wosk independently ami in 
i^nall ^ups without constant siqiervisicHi. This means that the tester is a 
manager, setting tasks and delegating most of the record kce|Hng to the studmts. 
The t^^r keeps a brief outline record fn>m tte review occaskms, as a 
convonioit remimler bow far the process is with each stucfent and m a backup 
in case a sfudent^s p^^oUo is tort* Only by systematic reviewing ami record 
keq}ing can the teacher ensure that stiKlent ev^uation Is fair and thorough. 



EiicelMGoocLNfugiaalUBsati^a^ Sik^ oim^n^ can ob vk^sly be 
given msnefk^ vali^ and thM airf ^nmied m a^n^ if a 

quttmitative amimaty isregdi^i MdwmtafaiMty iRgposcs. Sitiithasiunnmsy 
woidd iKvcx v€^ i^ui as feecll^ 
Nimnative tm^pi€^^kH» idiotild be do^ 
c<unptet^ in comparisf^ to die jieA of the 
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Ths Asxsm«at of Grotq> tHscussbiis nnA Coti^^ex: Probtem 
Potential Contributioiis of Sckenm Tliecry 

PtiilipNagy 
The Ontario Instititte fw Su»it^ in Edw^itticMi 

IniToducticH 

The jmiposes of this rqm cratie cm tf» iffiue ctf 1k)w to as^^ 
m(»ecQmptexouicxm^ctfediK:MkHi. Tf^ttecsetical&wiemirfctste^intl^ 
Ik^attne cm the serving of iU-^nictured {mri^ms, in particulw the qiidiartion 
of «:tenia thec^ to framnig aini ocm^iaring cfifiC^i^ solutic^is to stK:h ptAh 
tens. Iccsi^idertwoexwildes: cHi^^iodivkliialsQlti^msof scdioolf^t^ 
to m i^iminl^mion fmridmi inesen^ as a ^udy (Nagy » 1 9^); and two» 
di^^isskHis mmng groi^ of elemeitt^ schocA childrw conconing a fiuniiy 
ccmflK:ti^ti^k)n»^»>pfe^tedasac^^y(Nagy, 19^). Tteccm^enis 
acfctn^sed gei^raliz^^ to the assessn^t many c^cmiptex edtK:^onal 
CHi&:(»ii^ indiKiing expository essays, gnnip discti^kMis, md any {Hot^tem 
solving situation whoe the notion c^a simple maiking scheme with one cotiect 
solitficm is inq^nnopriate. 

The ^lapeoftf^I^pea- is as follows. First, using the adminisuativeint^tem 
example, I discus oi^ variant of a metlKxi fcH* devek^g a tl^cm^caliy-l^Lsed 
fiamewcdc for comparing probtem scriuticms, and the pCMsibilities and dtfffcul* 
tks M^rent in the pnx^. Tb^, using the family {noblem examine, I Imefly 
examine amtfher vaiant finally, I leport the results of an attemfH to have a 
gfOiq> of expertenced teachers s^\y the coding system devek^)ed in the family 
proMem example, 

Tbeie aie two thresh in the paper, not entiieiy sq»rable: first, the de veJc^ 
ment of a coding system which will reveal differences s^ross individuals 
groups in problem solutions; second, at^ltcaticHi of that system by teachers. 

Bockgnmnd 

The literature related to this study comes from several fields of research: 
first, stc^e in memwy , in which interest has grown fnm memoBry for ronsense 
syliabk^ tothatf(»'mcHecomplexphencm^(KinL$ch, 1974); second,probiem 
solving, in which inters has grown beyond simple problems with clear-cut 
solittions to ill-structured patterns with cc»nptex s^uti(Hi$ (I^nederiksen, 19S4); 
and third, assessn^nt, in which it ^ increasingly recognized that die more 
complex goalsof education lack £4}propriate assessment methods (Archbald and 
Newman, 1988)* The summary that follows is an abbreviated version of a fuller 
discussion in Nagy (199(K)). 
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Schema Theory 

R^^ict^ have p(^^ ^tUi ^ ran^ sd^im^ 

(Andsson^ ^ Amtersoii. 1978} act as 

g^^ral know^ig^ ml me mom ebstmA than die particulars of a gimi 

sititflh»L Iiileii»?tati(m of m hidivi(httl*s B^nOTtes In t^ms of a i^ooia 

involves m^ingel^i^ts in the cbaafirc»n the 

placehcdd^s hi the schema F6r eximiple, tidcen fi^ 

ffimtm^ized below, m a skrt l^ted '"running a staff meeting% cUninent 

indivkluds wiU have difloent vi^ws or ^tegtes to amid ccnifiontmkm« 

g&^»^ discu^icHi, or buiW team f^^qmtion. 

Cognitive thecvi^ have ^»ted tte psycho&igical st^ ctf schemata. 
Abel^ (1981) aigues that ^diemaia have psydioh^cal reality, ratfi» thmt 
nwi^y being orgffliixcrs for the COTvenience of r^^airfi^ C^the<rfi©rlMKl, 
Alba asKi H^f^ (1983) argue that lAcved meniCHi^ 
selected subset iHedictedt^schanathecMry, Their perspective, that & might be 
aiqw^iatc to vfew Tirana thOTy as a mrthod c^tmpc^g 
ity, ncA necessmly invcrfving any strmg as^mjAicms comeming tl^ nature (tf 
human memcry, is acte{rted for tfie pncsCTt resefflch. 

As advice ft»* irtqiosing ortoandexaniining difleiences, schema theoiy 
hoUspromise- Fbrexampfe,Schdleft(I982)0otesthatsch«natabecomemoie 
elabc^ate and specific with ^pe^im^e. This suggests that an examinattra of the 
dmils of slory-Iines across individuals might be used to highlight difli»n»ices 
in ^)ecificity cnt sc^tsticati(»i, diffemices whk:h in turn might be linked to 
e?^rki^e sxid/m expotise, as in tte admini^tive pioblem di^ssed below, 
or to age or educaticni, as in the family pix^)Iem. 

Scl^mata can be generated at varying levels of ^)^ifidty. Returning to the 
example above, a sch^a irflacdirtkter at a v^ gc»ieral level might be "aalf 
relations**. Within that level might be several nKwe qieciftc dots, mcluding 
'•running a staff riK5cting^ Within the c^cwy , even more speciffc, might 
be "avoiding confrontation**, '^generating discussion**, or *lMiilding team 
partfcipatiOT**. Thai,withineacbof thosecmc^CMiesmighibesevcralpossibili- 
ties, tl^mselves diffmng in quality. DiffercnliaUcHis cm go cm indeHnitely 
until, in the extieroe, every datum has its own category. The point at which such 
iq)eciffcity produces us^ul views of the data depends on the purpose of the 
cxaminatkm. 

One of the many difficulties with such a pars^live is how to decide what 
cOTStituies a more complete version of a schema, i,e., making judgmrats <rf 
quality. Hortonand Mills (1984) cmcluded that the schemaat^[m^h isplagued 
by the lack of an independent deHni tton dTdei^h of pnoccsstng or ^histicaticHi. 
Thus, a pre^nt limitation to the techmque is reliance on sub^x^tive decisions 
cOTceming the val w (i.e., level of sophirtication, wcMlh) of particular peces of 
data. The coniKCtion between mennny research and problem solving consists 
in viewing solving of a problem as retrieving story elements from memory. 
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PrMemSaUdng 

Tte fxm oi thte is on *11{-aniaiff)^ protons (Rnecto&s^, 1984), 
if a sobiba faas bem iQ^j^ b»:k of 

m(W6g»»Blc»r^, wdnoGcmv^bml^of^^^ Ttey^soteve 
h^to veited ctmt^ aid aie mem ccmttP^t depofteiit Most ''Kal-Ufe** 
fm^jlms would be cl^sified^iU-^nsctu^ V(ma»dI\^(19S8)iiG4edthat 
tte n^hod cbo^ fior dnii m^ym erf* tU-^n^Stued pnHrfems reftects tl)^ 
tteOTBtkd commits ^ ifie tiiveAtgM»s. Thaw exmni^ deimm^rale tl^ 
vmiety of themi^ical ccHK^ems ami approaches used* 

L^kin (l9Wi), primarily ram^en^ wiUi teadiingt wt^iced in i^ysii^ 
ami als^Ha, meas wfudi exhibit son^ ch^iK^^i^Hrs ctf both u^-stiwtiDnQd 
acd ilt-stnKtuied pY4:4eiiis* She has found that I^ge-scale units $iK:h as 
Schaidc*s (1974) scrqHs, simile to sctemata, are useful in ttie analysis oi 
pn^desn solving in such dmiains. Voss, Greene, Ptm & Pmner (1983) set ((X 
their wb^cts tte psiMem of tte lack of (Hoch^itvhy of the SoviA agricultural 
system. Their main OMKOfn was understanding die probtem wiving jHXKe^ 
(Voss 1988). Tliey categorized ^atemems as gral f^tements^ which 

deal widi relatively global moves^ smh as it^tiflcation of issues aiKl 
sul^m^rfeflis, reasc^ing st^onents, which cfeal with tl^^ analysis within the 
stnicttffeoftl^sestrf^n^lems. Finally, Lawience (1988), also concenM^ with 
undexstaiKling of the im)blem solving proc^ (Vim mid Post, 198^, mnked in 
the context of jiKlicialcfecisicHinuddng. Herbaskanxfelcmmstsof ebdx^Bte 
if. then statemems. Ste spes^ cmsiifeiBble elTc»t on tt^ need fc^ m analysis 
system to capture a pAmi perspectives Of ram» of refon^ce**), which ccotc- 
spond, ^xoiding to Voss and ?osl (1988), to die magistrates' cmirtroom 
schemata. 

Voss md host's (1988) linking of mctboddogy to Uieoretical framewotk is 
g^mrnie. Tl^nuHivatingcofurerafcKrthepr^entstudyistoeximndthear^^ 
of assessment (fevices availabte si the school level Thus, when faced with die 
ctoice between richne^ of (fetail and operational simpf ificatton, the latt^ must 
be chosen. 

Assessment 

There is considerable dissatisfaction widi the imi^i of tnditional (i^e., 
muldple-chdice) ^dardts^ ^ng programs cm ^hool cuirk^uia. Nagy, 
Traub and M£K:Jlury (1986), in a review of this literature, point out Uie danger 
that wfmt is most easily asse^edtmis to become most important Atthesame 
time, tha:e is a movement toward the t^cbing of "higher-OTder** thinking 
(Resnick and Kk^fer, 1989), Despite |»ogress in assessment metlKxls (e.g.« 
NkikerscHt, 1989), d^re is some ants^onism between the leaching and as^ss- 
nientctfhigherH^rperformance am! traditknial standardly Calls for 
improvement in ass^sment (Hae^tel, 1986; An^bald atHi Newman, 1988; 
Stiggins, 1988) toid to be calls for devek>|m^t of technolq^ies beyond the 
multiple-choice item. In addition, thoie is a wealdi of evidence that much 
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tester level Maifiy<^tteaitk:teskAiMfea(m(I5^)<lQCimi^ 



The Admimstrutiw ProWm 



Tte following is an at^viated r^xxrt cf a sUk^ mepc^ed in Nagy (19^)» 
Tte ni^lKxktogy* mb^Hed bcm Voss, Grec^, P^t md Pronei' (19S3)^ 
developed ova* sevc^iebitedstiKiies erf' tow FuU 
details of tte evdittkm dT Ite metlKxIdc^ are avail£d[)lc in Nagy^ Allis(Hi» 
Allison, & MoOThc^ (1990). 

The sowiy was an sunaiysls of the responses of 31 pf:e»:tising etemeitfaiy 
school pdiKipais to a cax sUKiy invcriving OHiflict b^wecn a sclraol staff and 
the school librariffli. The situaticm involves e^nents of »ip^tel(»i, curricih 
lum, pdicy» intecpeisc^ lelatbns, {7hy^::id[ i&m, budg&t and RQiply » and ^aff 
attitiKtes. The sitimtion was pn^^ted to the object fOTSolutk^ni^ 
a pdncipai new to tt^ school 

Subj^nsw^trabed in the think aloud proce^ and were asked to read tt^ 
case abud, interjecting ti^ ttKHi^hts as ttey lead. Then ttk^ were asked to 
think akntd akcmi how tl^y w^utd bcAvc tl^ {KoUem« and finally to recall tteir 
ttought {Hocesses. We ain^ (or eight subjects from e^h of four experioice 
groups, A^irant (0 years). Novice (1-2 years). Seasoned (10-15 yeais) ami 
Veteran (20 plus years): di^ to equipuKitt maifuncti we ended up with data 
from only seven Seas(»^ princijpals. 

Amdysis and Res^dts 

The c^nce of the irocedure used to analyze the data was to buiki a 
collective story-line, across subjects, capturing the variety of responses to the 
prcM^, inchKiing valt^ exhibited, p^:5pectives taken, and actions planned. 
Within this collective framework, then, individual responses to the problem 
were highligjited mfi ccmi ored. Table 1 gives a simplirted version of the 
schema built from the collective responses the 31 subjects, alcnig with the 
percentage of the statements that fell into e%h categc^. Briefly, subjects dealt 
with the probl^ largely as one invdving the libiarian, the tibiary , ami ll^ staff 
(Cab^ories S, 6, and 7)» They talked about the prcH>lem solving process itself 
(Categc^ 1) largely becau^ they were specifically asked to, and talked about 
context (Categories 2, 3, and 4) relatively little. The reasons for ordering the 
categories as in Table 1 are discussed bebw. 



Table I 

Stultified Sdiema for Ponrayat <^So{ucU»is to Adminimative Ph>blm 



1. PROBLEM SOLVING (17%) 

LI DdinituMi of tte ptrirf^ 

2. COMMUNITY (1%) 

2.1 Seek ccHnmunity and ^udent input 

3.SYSTBM(2%) 

3.1 Ask about brard policy ot pixx^edure 

3.2 CimsttU with colleague 

33 Bring in library resource 

34 Bring in personr^ re«nirce 

4. SCHOOL (3%) 

4*1 Sctool goals 

42 Atmo^cre of school 

43 Timetabling 
4.4 Vicc-Piincii^ 

5. THE LIBRARY (25%) 

5«1 Role of the library 
5*2 PtescntprKtice 
53 Iwprovdwnt 

6. PERSONNEL (22%) 

6.1 Entry, data gathering and rq)pcHl 

6.2 Ccmflkt, trust 
63 Staff meetings 

6A Staff develc^ment and supervision 

7. LIBRARIAN (30%) 

7.1 Diagnosis 

7.2 Data collection 
73 Transfer resolution 
7.4 Suppwt 



The full category system is more (fetailed (see Nagy, 1990a). It contains 7 
slots fc^ data al the second decimal levels 20 at tf^ thirdt 47 at the fourtht and 12 
at the fifth. It is passible to> in effect, start at the top of the full category system 
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ami a ^iGfy giving 8 simiiii^ (tf what evei]^^ 
respcm^ to tte case. Smdi a ^(Hy4ii^ cqqiem m tf^ cmgii^ 

U^ng this syst^ l» (Kgmiize ffi^ tafn^tte 
of molts were fotaicL F!i^ ft iiw pc^^te to isdMe (Bfi€9n^»:es 

stat^nents com^emkg the iiiqx»t»K:e ctf 

mlyexper^K^inm^il^bgPvefiserj)^ bi^Kifirast 
(pari^), tfK)se tess oqiedc^M^ 

of im>vtecial Bfasary &c(m!, weiaesmtod ^iimiraiesitf 

^^knis takra tte sub^<^ £0 a i^md 

ctfediK^^kMidl^miiriArAkHi* lltese experts gave quite (^msbtentnUiogscMi a 
10-poiirt scale, with wfiKh we isolal^ the five 

tte)^ Seasc^ml) aid the five most tow n&sd (three Novices and two V^^^ns) 
^tttions* Again, we fcHind^ieciikdiffeim^FOTexampk, the high sc^^ 
sub^ts consistently scrficited tf^ p^smal feelings of tt^ Ut^arian and helped 
hCTtocfeveIq>owna:^ip(tfherpn^4enis. Fhially.wewefeal^toccmipaiethe 
diflerences between e?i;pedm^ and inexpedefK?ed people with iIk^ betwera 
high and low »^ng pec^Ie. We fcHiml smie dw^K:t^stics of highly lated 
sduticHis that cmcdi^ with experience, atid others with youth, at leaft lack 
of expoi^K:^. For example, bcMh th(^ with high experience and high ratings 
talked of jriffiining bcforetmnd fear stafT meeting, and havii^ infonnatira 
avafli^te. In connast, those with high ratings and those with low expedence 
talked of setting pricmiies (i.e., mbjHoblems within the largca- problem) and erf 
{Hoviding professional dcvelc^Mront (or the librarian in both litKaiy and inter- 
per$(HiaI skills. 

Commentary 

Several concqptual and metboddogk^al {Hoblems which anose in the data 
analysis are cent^ to the pre^t discussicni. If we are to regularize siK:h a 
process fcr the sy^ematk: analysis of complex vertml or written output, thOT the 
procedures ne^ to be simple, agreed upon, and consistent. If we are to furtha 
put the techniqiK^ in te^hm* hands, it must be lime-enicient It must also be 
rewarding to the individual teacher, in the sense of producing infcHnuuion useful 
for the instriKrticMial proce^. Given the lealities of inesoit student grmfing 
systems, it must also be in ^me seo^ quantifiable; this If ^t point, however, is 
not toudied upcm in this paper. 

The initiaiiy eiKi^ounter^ isst^ was segmentation of the protocol into units 
for analysis. Ilie gral is to isolate 'Ihoi^ units'* from each other. However, 
detmninationof when ch^ thought tmitends and another be^ns (in effect, when 
the subject chmges tq>ics) dqieiuis on striking a balance betwmi capturing 
d^landproducingamanageablecategorysystem. The problem is made mcne 
complex by the ill-strucuired nature of tl» field; tte categcsy system crfn^cessity 
evolves as you work. Technically, by using a w(H^d proces^ng system with an 
automatic paragraph numbering feature, it was possible to adjust the protocol 
segmentatim to the evolving category system. However, an inherent difficulty 
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in dm s^meiitatkin pnx^ c(»K:^m faictividis^ 
c^^my, (Xto^ooi^b^i/i^ra the analysis 

ti^ ^me l^igth ut^rance. Tbcchoic^fk^by ll^^yaaretocodeaIar:i^ 
veii»!oitfi^8S(^iBitt,(srtoartilfe IfsegmcattaikHiisduK^ 
mi segm^ts l^^eDed diflfamtly firom each cnte*, ttea this is acwjmpli^tied 
only by iotroducing iiKiea^Iy fima' but less tisd^ tevds of detail ii^ the 
csas^osy sy^eou If segnraitatoi is avoided* thra speedli imits 
diffo^ te&gths mi om^^i^ £^ up with the sane code. Eitiier 
results in both gains md losses. 

TI^ scccsyA issi^ 1^ orgamzatkHi (rf the categc^ s^t^« Initially, 
ationpt was nuute to use '"^^tknis teken"^ as tte (organizing ]xiiK^ii^ Tteie 
weie» however, too many sli^tly different Ktkms th^ to be feasibte. SiK:h 
a ^a£m wiHsId have leailted in huge numbar^ (tf calef^ 
iqieciftcity. In^e^ the|^iKni|^ tmed was ma^ait^ into »ib-aieas, 
fallowed by detailed »:tk»is within d^mib-ffi^ As T^e 1 dii^teys, seven 
ma^ aieas feU (»it (tf the data. Many categcxdzaticms, even imo these sevoi 
bo^ge areas, wore difficult declaims. To minimize this iHxH>fem, analysis 
IHOceeded fium the m(He gi(^ to tte imm ^Mcific ai^ 
in Table!. Firrt^ staten^its on tf!ep«*lem saving irocess^^ 
cat^(»riz^, then on the (immunity, and fnially cm the It^^amoie 
piecise aiKt tess taxing t^ topo^pcH^ c^egorkaticm only when a ittatement 
cte^y fit a lanallar c^g(Hy th^ topos^xHie wl^ the ^^^n^t only fita larger 
category. Ot^ within the sev^ larger categoric, ctevelofment of the sub- 
categmy ^ructure was a relatively draightforw^ task. 

Tbethiidis^isthatoftevelofdetaiL Tt^analy^IHodiK^ofdifTeraKes 
ts^at:^ princij^ls in ap[TO^ to the case leveals seme int^esting differences 
related to experiei^ and rated quality. Many of these offer food for tlK)ught 
However, despite the fii^grainsd aimiysis, nune detail frc»n ti^ tnu^cripts 
woubt be welcon^ in nunim)u$ instam^. Further woit is required (aiKl 
schedutMO to complete the miysis of the transcript data. To illustrate die type 
of analysis required, consider one el^ent of the schema at a mote (tetailed level 
than TaMel. Category 6 J entry and familiarizaticm^tegie& Inthisinitial 
analysis, 48 i^tements in thb categc^ were ma^te by 20 different individuals. 
We have b^en these ^tements into two ^nailer categories. 6.1.2 J, persmal 
knowtedge and 6. 1 22, professional kwwiedge* but we have made no attempt 
to assess the variety quality of treatm^ts and suggestions within these 
breakdowns. To do this without prod(K:ing an unmanageable coding system is 
a major undertaking. 

In summary, the analysis is clearly successful. The coltective sch^na 
ai^Hoach to protocol analysis is raaimgeable, even with very large data sets and 
very cinnpl^ pn^Iems. The results i^w interi»etable diRierc^ces betwera 
inexpedenced and experienced principals. As well, they ca^t be relied in a 
sensible mani^r to indq)en(fent judgments of reqxmse quality. Further, it has 
been possible to identify elements of good problem solving whidi are related to 



^qped^sx cMtms di tbx^ teas «|«ieiiced ss&n ta possess. The 
listiucticmal ta|rfk:atkH» sudi a t^nk rcquii^ miK^h ffstter wcHk, but k 

ekfa^ our c^egGfy system or n^hod oi s^jarating bmiscx^rts into s^pi^ls« 
We (to m>t kmm iKm or wh^^ we rag^ 

^^OQ^ givra by leqxHHtoits. P^rh^ nmt difffeidt to ovciame* the 
r^x»ted£iii^iy^bi^cc»uwi^ Ifaiiythmg is to omectf this 

tl^ is both thecai^kaity \mGd and u^ul witfun the ron^naints of classrocm 
tiie, imne w(nfc is required. 

The Fanuly Problem 




Intwdu^ion 

The following is ao ^^Meviated version of a Sudy reported in more detail in 
Nagy (1990b). The purpose of the sUKfy was to an dialysis of discus- 
sions, flnicmggnnips of eton^tarysclmrfcMl^^ As in 
the ^faninistrativeim^^em ^xyve» theioal^ nH;tlK)d used wasan ^l^pt^icMi 
of schema theory s^ against a t^kground dfrecrat research <m die solving d 
itt-strwtured problenis. The {^rticulm* data analyzed came fnmi an cmgoingarai 
much larger currfeulum proj^t c(Hni»ring metlKxls of teaching thinking skills 
inttecIassnxHn* llieccmtext$^esasavchiclefordi%u$sionofn^thodok)gi- 
cal i^^ ntfher thanactefinitive t^d* the ciflnrK^ula tmder study: Vtsst^ tl^ data 
can^ fitmi the end of )^ om of a three-year tHX)ject; and seccHid, the treatments 
were sid)ject to a great many de^d vagmes. 

Tte subjects can^ frcmi dghtniral dm^tary scl^^ 
jurisdictkins in Sknidiwestm Ontario. Nine sdiools were pan of the larg^ 
study; sdfiechiling pn^ems i»evrated data colleokm in cme school. Th^ nine 
schools were assigned to CHie of three tneatmoits, and within each sctool oi^ 
Chmie 3and(nieGiade6ctass werechi^oi fOTpetfticfi^m^ TocoUectthe 
discussion data, ^mlcnts were tak^ from class in grou{» of about Ave and asked 
to discuss the pobtem (m ten minutes. Acn^ the two grades and three 
treatments^ 76 siKJh discussions were recOTcted. For jrosentpirposes, details erf 
tl^ Ueatments are mA required; they will be refmed to as Experime»ital, 
SuppOTted, and COTtrol 

The focus of this study was an "ill-structured** prc*lem. Groups of students 
were taken fnrni class and presented, both cnally 3nd in writing, with tbo 
foUowingsttuatk)n(n(^ that ndtl^r age iK)r gender is ^)ecified): "^ereare 
two diildrra in the Puzzlewkh family. One child is called and the mh^ is 
BJ* Both of the children receive tte same allowaiK^e* Ptt is involved in many 
after school activities siK:h as music le^ms, ringelte, church choir, and youth 
group. BJm however, just attends youth group OTce a wedc, Mrs, Puzzlewicb 
is always asking B J. to cto extra chores around the house. She NEVER asks Piat 
to help out 6 J. complains to the mother that it is unfair to have to do all of the 
chcwres and yet receive tte ^me allowance as Pat. *I want an im^rease in 
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dkiwaoce/ TteMod^ss^ *Youffieo*t paid for dKHes^Ycmr^hi^ 
pm. for bw^ imn d this &mUy . Ym may not bave m imse^ in aflowamx^. * 
Whatdoyoutliiiik?'' 

Asudysis mdResuUs 

Bdiran^ otitlinmg tte ffi[uily> !3 aiKl re^^ 
b^we»i tliis ^miy ffld ttat Bivc^ving the pdi^^ 

a cate^csy sy^em Uff tte chihtira d^xhi^ a .ntKdi mcaie difilciilt mKl 

1^ stfi^V^ Tl^ woe tiro mffl0 One was a g^ieial 

lekK^t^Bcetoiaflq fai sm^c^ltecfiscsBwms 

to rmaio as passive a3 pcM^Me, talked as mwli 

S^:m^ nw? h (tf tte d^uis^ 1^ unf ocu^ 

r^i»n»»:e to previcHis qieak^ oul ia^ 

diffiodt to be certain erf* tte pc^ t^ 

Tiie m&bxxk)iogy ev(^ed dtaing tte aoalysb, eventuaUy imxliK^ing two 
pRxhicts* TbQ fmt was a melfKxl of tca^fa^ the (fegiee of cohc^m in tte 
dtscussicm — thecsitenttowhkb itwasacvrnvcmtkmammgdiegn^nttb^ 
than five chtkix^tidcingUirnstaUu^ Brtefty»tfitswaspartialty 
succe$sfiriandiHtmiistng«biUwiUm^ Tt^secxMidpnoduct 
wasa two4eveI c^gCHy systmi f(Mr ttestat^nents made^c^^ 
basic aritective sci^a with reflect to family fatniess. Unlike tlw jKincqmls* 
sttimtk>nt it was not possiMe to cr»te a ^iny-line. Esch cat^ory devekqied is 
a kxiwly held togetl^ (^Itecticm (tf sub-categories that deal with roughly the 
same pc3'^)eaive cm the issue. 

Tl» cat^ory (tevek^oit jmicess is interesting. The analysis began with 
ad hoc devek)pnient of m elaboraie categwy system f(^ the steten^nts made. 
C^ice cc^i{^te» it was cteaned by fenx>val of semal categcKi^ tf^ 
INnpc^, usually di^ to 1^ of fii^uency. Hiese inclucfed c(^nectives» 
^atem^ of facts and assumptims ^>out tti^ c^» humour and fantasy* 
incdi^ence k sdf- ccmtn^ticm, mnt cramients m the irogress of tl^ 
discussiim. Next, s(»ne v^ g^^rat ciO^CHies w^ created to keep tlu^ data 
simple. These inducted inxMnpti fimi the int^iew^^ g^ral agreement^ 
sp^fK agr^ment, peisiH^ ai^c^k^, and details* which weie usually ex- 
l^iHiing cm a point beyond a l^vtl judged useful f(x Urn mt^ided analysis. 

At this stage* we about 100 cathodes in five large giou{^ unfairness 
statennrnts* pn^>osed solutions* c^(»is dbcmt i»roposed soluticms* ccmiments 
on ti^ relevance of age* mn! vahie positicms. Several fH^Iems became 
^ptfent' the system was un wiel dy * many of the cat^cnies were used only oaco 
or twke* many c^mired v^ subtte distmctims in meaning* and bound^es 
among the categories were unclear. Con^d^trfe collapsing and reananging 
was bc^ nec^sary and relatively easy. A second sorting* widi some amalga- 
mation and deletim* yielded six FK^iticm categories and seven Action categi^ies 
as listed in Tabled 
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TabfeZ 

Otfegc^ system the femity poblem 



(I) Posbion Statements 

Fc^tkni*! — statxaneirts stowing sccq^ance of lespcmsibility ta^ 
ami airamie^ of tli^ imacfer fomii^ 

PicisiticHi"2 — ^^onents stowing a disie^Btl (m iei^[xmtbiUties, iiK:liMi- 
ing j^atonents that Piat oi^t to imip only wii^ amvmient; 

Position-3 — st^^ents that tto family ou^ht to operate on a monetaiy 
basis; 

PDsition-4 — statements stowing awareness of age, and its impact 

Position-5 — ^atem^ts about tto feelings of anycme in the family; 

Positson-6 — statem^ts that both chones and extracurmutar activities 
have value for ^ individual engaged in them; 

(II) Action Statemmts 

Action- 1 — soluticms which involve differential allocation <rf allowance 
or balance between the story chai^tcrs of activitieSt chores* 
andrewaids: 

Action-2 — soluticms which involve thieving, by a variety oi means, a 
balance betwe^ the story charactm of ^tivities* chores, and 
rewards; 

Action-3 — weaken solutions involving fairness when convwiient; 

Aclion-4 — solutions involving unilateral ^tion by B J,; 

Action-5 — soIuticHis which involve emphasis on a process, such as 
discussion or keeping records, or setting up a schedule: 

Action-6 — more respOTsible solutions involving family coopemticHi and 
sharing costs; 

Action-7 — a catch-all for unlikely or irrelevant proposals. 



Z Tifvo f(»- suteequett analysis see^ 
oiriciiecl, tyi^cal and ii^ 

mKlexmmimi^tl0ri»acn^gG^esa]»iti)e^^ EvoiacuiscMyraffiiifii^^ 
erf 0^ <tei»»^ia£ed tto fim path wimU be 
Xii^ the diiftdra cmssxi^ was noi cn^^t ^migh within a single 
di^rtisskm. TtefiBfmB,Uielatt^I»th wi^cfM^eiu 

Fc^ toitial purpcm tte thirteen ^tement catq^oiies weie objectively 
to quality as foUcws: 

Position Actk)n 
Typical 1 1,2 

Impoveristed 2, 3 3, 7 
enriched 4,5,6 4,5,6 

Aiqmixtmately 72% of tbs Giade 3 le^xmses skI 61% of die Giade 6 
re^xmses woe captuied in die Typk^al Scl^mata. Hie coneqionding figures 
fwdw Impovcridied Schemata were 11% (GnKte 3) and 13% (Gra<te6); and for 
die Enricl^ Schemata, 17% (Gr^ 3) and 26% (Grade 6). One obvious 
ilTw^ty an^ concCTiing die saibjpoivity of d» systan- Two of dK vtews 
dubbed Impoverished wore more common among Grade 6 Uian Grade 3 
stiKfents, asui on reflectic^ are best explained by imrcased self-centredi^^ss 
resulting from dieappix)^ of adolescence. The tentative and subjective nature 
of diesecat^rizaticMis ought not be overlo(*ed. 

Since Typical ^tements were so much moie fiequmt than Impoverished or 
Enriched statements, difli^^t analyses were lequii^ Fot the TyiA^al state- 
n^tSt bodi freqi^ncy md v^lety wiUiin tt^ Uuee lar^ categories were 
examined (w each discumni. Note thM die 13 c^^cmes in Table 2 wero 
{HYxhK^ frmt more dian 100 sniallar categcdes. Fac the Enacts and 
Impoverished Categories, simple occurrence as a function of grade and treat- 
ment (Expoin^tal, Su{^x»ted, and Qmtnd) was ^^ined. 

Tb^ weft diflfeieiK^es ^ross grotq^ in the extoit to which diey expressed 
sentiments dubbed part of die Typical view of fisunily fairness. Ttese diflcr- 
eiK^es, towevCT, did not fall into a simple patt^; giv^ die circumstances, 
systematic differwices were not expected. In die EnricNtd and Impoverished 
categories, grsKle difT^races were quite cmnpelling. The Grade 6 students 
made imqxHtionaliy mcHe than twice as many df the Enriched Schemata 
statements, as would be expected, hit, ^ mc^tiimed, diey also made more of the 
Impoverished Schemata statements. Tt^re wc»e as well seme discernible, 
although unsystematic, treatment effects. 

Commentary 

This dialysis rests on some assumpUcHis concerning the nature of the data 
which need to be discussed In particular, we need to address the relatimship 
between individual and group data. Schrag (1988) has argued that diere is no 



way ofasKSsing the (hinldjig required forat^ 

in tl^ case bdweea B J. aiKl Pu's finnUy proWon »td the tec^^ 
This infooM^ difHralty B a ocmuncnqdac^ 

wiKHBtt iiifexm» in cogiiitfve r^e^ch. &i ti» {H^sent ca», or.e needs to 
ffixept il^ geiKi;dly the tyi»cai ^^maits are an ^K^ir^ 

ttttkm of tte typk»I tt»tettt*s th^ing. 

Pnfi^ nuire coiiKitkMis is ti» accQMtmt^ tint in paiticuta- eiKh 
^)eech ^iequately repre^t^ hi^Iiar thought processes. That is, what effect 
did the gnnq) t»ve cm tiie dnHty (tf inliviibial to thiidc (NH 
express an opinicBi? There is no evidoKe avail^Ic on the question of whether 
seme, indivkluals felt compelled to eitlur remain silent (x p^ve 
agreement wfa^ £iced with tte expressed (^nkms oi more a^ertive cbss- 
mates. HiisisareallinutmicHitotfaedataavailidtle. In tl^c^iginal repent, the 
ttgun^ was pm f(X^waid ttat with Ave AiKtents per grmq), a discQ^^ 
of 10 (that is. ten changes of ^leaker exchiding the intoviewer) means that the 
avei^studaitlo(^advwt^oftheq)pcKtimity to ^pe^ twice. Seven ofthe 
76 dii»»£Ssi(His were ^hoftra than 10, while 57 were kmga- than 20. "Wbitetl^ 
are iK> data m individual b^^iom- withm the gn^q), it seems safe to oonchide 
that, while «Mne stiKlents might have beat undidy reticent, a sutetoitial 
nuycHity {m^ly tocA the (^jportunity to exixess their views" (Nagy, 19^). 

Om diOkuIty of die metlKxl, idoitifled in die literature (Hmton said Mils, 
1984), is tfiat the categwizalkMi of ^en^ts is at itxM aib^tive. What wie 
would like to consider as a deeper level of processing could as easily be 
canstnffid sim|rfy as mwe like tt^ smtiments adults wcHiid like to see chifaSren 
exiHess, TheproWem witfi thisfeatiaieof themeUiod isdemraistratedby the fact 
that the Grvie 3 stt^oits af^iesedless elfish, in a^iectsof thekiHtttoc(^, dum 
theGrade6studaits. C^mjightchoc^e to define growth in pace;Hion of die 
situad<ni empirically by ac(^ng whm might be a natural (»itcome of adoles- 
c^tice. Or, one might choose to cc»i^(^ what is desirabte fnan the «lult 
perspective as a valid ctoricular goal, aiKl take the Gr^ 6 resultsas undesirable. 
The inublem remains, however, of having to di^guish level of nM»aI devel- 
opmtA (however defin»I) frwn fevd of cognitive im)cessing. 

A sGfXXui difficulty of method is that, if we are going to examine very ill- 
^ructured pffoblems with no inherently obvious "correct answers", the categwy 
system must evolve from lt» data. Where one »aits in die data analysis is 
imp(»rtam. When to open a new category is an arbitrary decisicm. based on a 
subjectii^ view of die history of Uieanalysis. Wfcitev©r''catego«y widUi" might 
n»ffli in diis context, effwt reeds to be spent in holding it stmicwhat cwistant 
lied in with die obvioiK issue of simple intor-iat^ reliability, already men- 
tioned, it would seem important for different analysts to analyze die data in 
dUTer^t (mkn^. The issue is somewhat simplified by the possibility tlmt an 
already-created <aiegoiy system, from an earlier investigadon widi different 
children, might be imposed on die data, but diere still remains die difficulty of 
valid and reliable creati(Mi of Uiat first set 



It is l^pbfaiia^ ta a& 1^ te» 
siJ^im tiKsray mi^ haw bera dbae fimi a mc^ 

ontef oobXH&es of instivct&m (Haestel. 1986; StjggiaSp 19S8)« Ndtfier the 

afifmiiHi^ oitfidi^ sgr^ems Both roqiiiie (tenlcpit^ 

io^xfikkm cf a tl»»mi(»l firao^^ 

ta^ Secoml, tftoe are oilb £nmi tS«m mSying itm assessmem practice erf 
te8d^(o.g^$tlggtosandBri49ef^ A 
ii^iliodalkniriivc(Kni»(i^ wUh a well-devekqpod imageoT what 

might be qtmltfies as a foe us^ metltod, berth for leseatcb mKl fw the impro ve- 
mentofp^tice* 

Teacher AppUcation of a Cate^qry System 

lf%troducHon and Method 

A msil investigatKm was coiulwted to see tK>w well a swipie of teachers 
wouhi be able to Bppl y ti^ category sy^im in Tat^ 2« Using a sy^emmk? search 
process thnnigh tte data of tt^ diikir^'s tiarocribcd discu^kms* 40 
tfatemms ¥^ selected These ijK:luded thiee Raiments initiaUyc^ 
belonging to of the 13 categc^ks, i^us an additional ^atcment fincm the 
final Action categcsy to px>dtK:e an evc^ number of statmoits. An instrument 
was put together consisting of tte following: (a) an introduction and thaidc-you; 
(b) the isigiimi stimulus and the 13 categ(»ies (plus a I4th ''(&)es lUA fit" 
cBieff^, (c) in»nicti<»ts with ck^ exfrfained o^an^rfe; (d) tii^ 40 statements; 
aml(e)aplacefOTC(»nn^ts* F(»^ tiunee ctf the 40 $tatenients> ex^eipts fro 
precedii^discusskffi weieincluc^toim^videmcNfec^ This 
instnm^ was distnbuted toa s^ple (rf (^veniei^^c^ lOel^i^ntary school 
tead^ cm one staff. Utey were asiced to woik individually, at hcKoe if the^ 
wij^l^. Nine of the group, including om stmlent teacher, returned the fmn. 

Results 

A comm^it is in order before rqx>rting the results, Wl^ I began marching 
the data base to ccmstriK^ the questionnaire, I found myself qm^oning many 
of my own initial categcmzationst and fighting the toK^K^y to searc^h (w more 
clearcut examines. This reflects tl^ difficulty of the categcHization task« 

Ute nii^ teacher repeated requiring an average of 50 minutes to complete 
the task. Most rqpcMted tte ta^ to be difTscuU. ard i^ve comments that fall into 
Ave general categories: 

(a) difncttlQr in distinguishing positic»is from acticHis; 

(b) tnsuffk:ient rantext or infcmiaticm; 

(c) too many categc^es with ambiguous definitic^; 

(d) difficulty in umicrstanding what the student's point was; 

(e) m(He than one point to the student statement 



CoQUi^its (a), (c) ffiul (d^ weie c»^^ 

af tt^ origiiml ^tfetnmts hsd been coded as dealing with mom dian one 
ctiefCHy, tl^se ex^i^ ted hem deJib^tely ^eluded kk tfie seleai(si 
proce^ 

Theresidtswefeq^ectaculariydissi^K^ R»rcmly7ofthe40staten^fits 
didani^jc»1ly ctf tteteach^agi^ with ^ 

none a^i^. Table 3a di^^ ti^ iMuie the dis^reemeMs. Hie mo^ 
serious (fifrK:alties with Actlcm Cstf^oii^ 2, 3^ 6p and 7. Positicsi 
Cat^c»r^Sffid68iidActicHi-4gavetheIe^difrK^^ Ingen^,tteniost 
CQEDmon i^t^Iems woe in changing jx^tic^ ^ateme^ts to acticms dements, 
am! in coding Ktkm statements as differoit sK^tions. 
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Summary of Agreements 



L With Inttfat Categorizations 



(a) Using 13 Categc^cs 

Agreement with OTigina) 
Disagreenrat within Positions 
Code Position Acticm 
Disagreem^t within Actions 
Co6& Acticsi as Positions 
Code as **do» not fit" 



--29% 
"5% 

"28% 
"13% 
--5% 



(b) Using 3 Categories 

Agreement with origitial 
Switching Typical and Impovoished 
Switching Typical and Enriched 
Switching Enridicd and Impoverished 
Codcas^^doesnotfit"* 



"45% 
--14% 
"19% 
"15% 

"5% 



With M^orlty Opinion 



(c) Using 13 Categories 

Agreem^t with original 
Disagreen^t within Positions 
Code Positicm as Action 
Disagreement within Actions 
Code Actic^ as Position 
Code as "doe« not fif* 



--45% 

"12% 
"20% 
"13% 
"6% 



(d) Using3Catcg«nics 

Agreement with original 
Switching Typical and Impoverished 
Switching Enriched and Impoverished 
Code as '^does not fif' 



"58% 
- - 13% 
"12% 
-^6% 



Tti^^tessffi^xraiigbig ways to €9(009^ HrM»Icritqisedtiie 

Inqioverii^^ Th^ 0¥ctU i^re^iic^ friHn 29% to cn^ Thus, 
l&tte (tf (to cfis^^ie^ii^ was betw^ cat^c^^ 

qml^. Second, sinM a ma^ pn^m was distinginstiiiig astkms from 
^^[^ tbisdistim^kin, teavuig cmi 

a^tn^)ov^id^ 11itsimisovedovc»all4^era»tt(r^^3b)to45%,tAtt 
^ lS%(tf ttereqxmses w^dis^green^b^w^ tfietwoextiemecmegories 
of Enrid^ bapc^eridi^. 

The next step was to mnove f nmi tl^ cuigitml categnizAkms thdr pv^ 
einbient ^atus, fflid to omskto'thmn as ni^y (tf ten 
such a Aance lesdtod in a c^ge c^categc^ for 1 ^ 
8tatCTiiCTtwasreaxied**&)CTmrtfU^thiTOPteit^^ 
Actkm ^d om ActicHi to Portion, mi ebvoi Acticm statem^tts were teas- 
dgrod to cttfaer Actian Categcn:^ Crngf^ Actkin-6 di^ppeaied oitirelyt 
abscsbedinto Actkm*l. As(^beserainTid>le3c^th^ unproved agreen^t 
c<»isfdenUy, from unacc^Aable to ^ im^TO^HaUe. What wo grmp by 
Typical^ Impoveridied ami Enriched, but ^1 maintain the PicmtkHi - AOioi 
disdiK^tkm, agre^noit rises to 49%; wbra we al^iKton itm PositicHi - Acttcm 
distinctk»i,itbec<mies58%(Tabte3d)« Thiswouldhavetobecb^edasbarely 
accq)tabte, but the pn^^lon of switches between the extremes crf^ Enriched and 
Impoverished still remains at 12%. 

Commentary 

On an optimistic note, it can safdy be assumed that mors of statement 
classtfk^Uon aie ramfomly distributed. The data cm fn^i^icy of occunence oS 
varioi^ ^atoment types, as in Nagy (199CMb), likely pointing in tl^ right 
directiiHi,atbeitwithrtfharwi<feccmnd@^baiK£5. Caiegmydeftniticntscanbc 
tightenfid, nu^ exami^ givc^, and training, or at leatt discussim <^ tf^ 
ii^tnK^ticms, pro vi(tedL One teacher in the sample provided an alt^native set of 
c^gorbs: awareness of (Hhers, s^Mnt^iested, valura woit , ^ws responsibil- 
ity, ndues money. Broactef, more abstri»:tedcateg(^es might be easi^ to ^ly 
with cmsisteiK^y than ttose used in this study. 

The diflfk^uhy of lack of context is a fault of the questionnaire inssrun^ta- 
tkm, mi would not be ajirobtem in a situaticHi involving full tian^i»s (x live 
obs^ation. Little can be dcm about tf^ vagueness with which diildien express 
themselves, especially what the extcmt of this vagueness might be an ob^t of 
investigation. 

All of the difficultly encounteied are exaceri^ted by oxitexts with less 
structure. Indeed, reflecticm on the differaices baween tte two repcmed cases 
suggest that the term ^'ill-strucmred** might be too bn^ a categ(Hy as work of 
this nature moves from the cognitive science labcHatory into the classrocMn. 
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Om onpoxiam diflan»ice b^wera tte inindpal p^rfonnaiK:© m (teir 
jmH^n 8ik1 tte diik&t^ CHI tteiis was tfKtt 
OHild be (^ived &(»n die pdmij^ 

fiBmewi»i^fc»'sysieiimticc<un{mi^ In contra^ ih© 

Emily ^taad^mdMmMiMivettetAra^ OiiI&tmsawthe^ti^k08S&i^(»r 
tm£E^^mo^yttieiau^)»aiid<^e^ These 
^^ge^kms can be cal^^^ized in 5e vobI 

d^L Titts ^^p^ to be as mw^ a func^km of tsA as the ^ges and 
^)phistication of the swifd^. 

Tte fi^mentatkmctf {Him^ols into units is at erne level »nqdy a m^hcxhv 
k)gk»l pnMcan, bm at am>th^ tev^, it Ues ck^ to 
bySchmg(i988)* ifasab^tmakesanuttefanceofseveialstatenK^ts'l^ 
tlvn i^ieaimaUy »di i^ is linked in smne tnm^ with die {sevkms ^aate- 
moits. Thisisaninferm^«butthedegieeofe^<»rconrKtocew 
mate tte mfeiem^ dep^ids €»i Ikiw the category sy^rai (teals with the 
»jtt(^i^!tStaiidcmtKmcl< .sdytI^lteinthetr^ wMior they are 
$epiRttedbysevefBl^at^(mtsonaiKAba:tssi^)« ThesjrstemasdevekqiedfcB' 
the i»incii4ls* case does not deal with the linkages between cate^orfeed 
sts^^ents. 

In rctcospect, ilie categiny system thm emeiged fnm the family [mri>lem 
discm^ions wns allowed to grow umrhecked. Tte level of categcnizaticm 
attained was too detailed, and was eventually aiwulomd in the ii^vitable 
colla^ngofcat^cHjesiecpiifedtoniate This[Nt)btem 
dkl not arise in the 8cho(ri problem. lliedocisfCMicmlevelofd^lttal^fiBuur 
earfy in tte development (rftte categc^ systcmi* held up as "thi^ about right**. 
Whether this too is a function oi tte t^ics and ages of subjects, or is more 
dependoilcMi theexperie»ce of the analyst (the imncipal study wasd(M)e seconil^ 
is an op^ qi^on. 

The attemiH to have t^fm ai^Iy a categCHy system was instructive ifmit 
entirely miccessful. The indk:atl(ms are fairly clear that a waller number df 
more gk^ categories would have been mocewodcable. Howev^, the point erf 
this line of research is to develq> a tteic^retical basis f(H^ grading complex 
cducatbnal ouic<mtes. Th^ is some unresolved t&tsicm between what will 
woit in situations and what can be grounded in theory. Such issues i^ed 
U) be resolved if theproduct of ihislineofrosearchfs to more than thecommc^ 
sense of experience tethers. 
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Construction of Curriculum Relevant Tests hf Teachers and 

Experts 

Univ^^y of S^c^hewan 

Tern an importam imt of the ediictik»[ml enf^jnise, MiU^ aid 
Erickson (1985). io their oitnxhKitH^ to a giucte f(X plambig, ccmstm^Ung, 
acbnitti^ering, mA inUspn^g te»:her^ndde te^, iimtoiscored the rnifKMrnu^ 
of studfflt ^dng and Ite fi^t tt^ mtctitkm 
llKysttted: 

Pnj^^y m> a^)ect of ediK^atiim is mcHe talked 
tlun^i^it testing, llieie^ several reascHisfc^txHhtl^insufricienti^ 
aid the pocH- test coimnK:ti(m— time pressure* ina^qume OKisinicticm 
skills, and u)cc»iect ^gment ^xnit students' ^ility levels* (p. 5) 

It is mainly thrcHigh te^g, both fom^ and infcmrol, that teiK:hers cm 
(tet^mine the status ofthestiKients in tkircaie. EfTeoiveteft^hing require that 
instnicticHibemtbeievdatwhidi^udeiitscanbefier^ How do we detmntne 
tte c^Himal level of instmcticHial focus? To ^liess this question it must be 
lealized that testing and evaluati(»i aie especially impratant sim^ most instruc* 
tKHial ^tuaticms are such that a class, not an individual, of hetoogeneous 
knowtedge-t»se, at»Iitfes, motivatkms. and {m)cessing skills is the targ^ of 
imtruetic^ intovention. 

Before addressing the above qt^CH) let us ccmssd^ the components of 
instructim. InstnK^(»iisthepnx:essof|Ke$entingi^aniccoiit^tin 
ofknowledge,cMcc^,IHinciples,gei^ralizmiQM^ 
ing thp cunkular goals. In^nictu»i wcmld entail aiuilyzing tte curriculum, an 
CHganizmional pten piesentation, deciskms reg^retmg modes of ddivery 
taking inK> accotmt tl^ entry bdiavk»s of students, and assessment of student 
pogress at varkHis stages of in«nK:ticML R^idhawa(197I)i»esent^aviewof 
instn}cti(^ system speciflcally fc^ indivklualfzation. This system incorpo- 
rated an informatJOT jrocessing unit, TOTE, first iMx^xjsed by Millor, Galanler, 
and Pribram (1960). A brk^f outlii^ of this syst^ is giv^ in Fig. 1. 

Gagne mid Briggs (1979) proposed a systems ^^moach for instructional 
design. This system represents a ^cs of fourteen stages that begins with 
analysis of needs and goats tovrard eventual denKHistcalicHi that a proposed 
system of instnK:tk>n is successful in meeting the stated currkular goals. Hie 
mo^ important elemrat in instnK^dcHi is (fecisicHi-making. It is through a series 
of decistons that m instntctional plan is drawn and imfd^ented. Decisions 
cmnot be made in vacuum. Tt^ context of instructicmal ckxisicHis is varied aiKl 
multi'faceted. Defensible decisions can only be m;gde with reliable and valid 
information chi students, re^nirces, and f^ilitk^. Gathering such mformalicHi 
in an efficient mamier is the re^nsibility of junctional designers. Te^hm 
become key players in the instructiotml &sign (os their classes. Thus, it is 
imp^ti ve that teachers have the know-how for gatheriung the relevant infcHHia- 
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tkiQforfa^nK;ti(Hi^(teci^(H^ The forasctfthb papier is ratftecmi^tiKtkni, 
use, and cf feed veoess of teachg-made cuiriculuni relevant tests. For compara- 
tive putpcKcs tmdier-made tests are ojnipared and contr^ted with standardized 
tesl& Alst^^ntt^es for inipuviiig teacher-made fe^ TO 
aigacd that these are highly (fcpeaident upcm mmdatory and imimjved instnic- 
ikm^ (^qxmimities in tte pie-service le^^ 
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Figure 1. An insiruclional model for groi^ or individualized imtrucUon. 



Teachet'inade Tests 

Extent of Testing 

A typical teacher spends between lOto 15%of theinsliwUonal time on the 
assei^ment of student progr^ ami on dtagnc^c infonnation galt^ring (Carlhr^, 
1981; Newman & StalUngs, 1982X GulUckson (1982) found that 95% of the 
teachers he surveyed tested at least biweekly. A recent survey ccmducted in 
Alberta tndkaicd that evaltiation activities of a teach^ took up ^proximately 
25% of the instmctional time allocated to a sub^t (W* T, Rogers, personal 
ccHnmunk^atim* May 10, 1990). Instruments u^ for this ass^ment by 
teadiers are usually home-made. In cxmtmstt the results of standardized tests* 
which are admsnistemi annually and at most grade levels in tf^ U. S« ^hools, 
arc used infrequently (or instructional decisions by classroom teachers (Beck & 
Stetz. 1979; Hennessey; 1982; Stager & Green, 1984). 

Surveys of tethers have indicated that on the average between 40 to 50% 
(with a range of 0 - 1 00%) of the course grades of smdents are dependent on test 



semes (OttlUckm 1984b; Mcfcee & Nfaoning-Cuitis^ 1982; Newman & 
StaUings, 1982). 

Comparative Personnel Support 

Te&sA^ m Can^tffl sdiods ^ iKtt have as miMih assi^^K^ and tssdng 
available as in mo^UjS.sdiocd& Most U.S. sctool di^ts have 
imxfe^^n^asmmattamlevahimkmpeifiKU^ Kbonly 
in a few larger sdiocd (Sstiicts io cmr ccMmiry that w& teve mxdk pcssfm^l 
pm^d^Rg^&^miccmims^mtQib^m^ This is not because teacters 
in CiBmlian »dioc^ have nec^^rfly betttf {»e-se^ 
meiit«evaii»tk}n,^$bttbtics. Furth^moie, the in-s»vk^ training 
(m ^s&im^ in thew meas nmy ncH be taken as the reawn mrt havii^ 
acfeqis^y tiaii^ lescmrce pa:$c»ii^ available ^ th^ sdrnd dittrict levd in 
nianyjiiri$dieti(»is* TheattitiKieseamtobeth^te^h^wiUscmieliowo:^^ 
todotte^ofassessanottamliiKleedti^yhave* Buttteqi^ityofftejc^dcme 
hasnotberaquestim^inmanyquaitecs. Thisnmybebecaiit^weinCaiwIa 
aie too poHte or that the iK:iKtenfc commtmity is afiakl (tf being cmtght in a 
vicKMis as to the re^xmibility fc^ my p^cdved <teficieiK:i^ in our 
t^Kdiera in this any other mea. I too wiU skirt this is^. Howe^Jwmtto 
faring to your attention CHie impoitwt fi^t from my own fi^mlty. Only in CNie (tf 
tte four m^jOTiHe*service teadi^educatkm{»x^$ramsincHira>lt^^ 
t^thieecmiit units in n^asuren^t^evaluatimi* In aUtte other int^rains 
^ncbnts qiend only stmt oi^-thini of tl^ time cm nK^asuen^ and evaltiation 
inattireecr^itclmonle^ingmutinstnic^ Onlyabcmtfivep^mitofthe 
pre-service tethers who padiatc from our college would have taken an 
el^ve statistics class* 

Teacher Knmdedge 4^ U$Hng and Measurement 

Hennessey (1982), Gullickson (1984), and Newman and Stallings (1982) 
sug^^ that tem^^' knowledge of te^ng techniques and their skill in 
classaroom testing practices was less than adequate. Since teacheis spcM a 
signi fk:ant im)portion of their instructional time cm testing gatb^g informa- 
tion foi instructkMial decisions, sub-q}timal practices may be detrimental to 
promoting excellence in learning and instriK^tion. There se^s to be consensus 
in research m the reasons behind this situadtitmt Some of tt^se are: inactequate 
pre-service and in-service educaiiOT of teacterr. negative attitudes toward 
testing on the part of administrates tethers, and indents; lack of motivation 
to Iram and appro{»iate roeastKement techniqi^; environmental context 
many £x:hooIs and districts; time ccmstiaints; choosing instructicmal options 
which do not rely mtK^h on measuremrat data; and so on* 

The number «rf purposes f« whfch tcKhofS administered tests and the 
nimiber of item typ^ they employed were related to the knowledge of measure- 
ment and evahtatton (Newman & Stallings, 1982). Tei&:^h^ with higher level 
of professed competence tended to use tests for more and SQ^i^te purpc^ 
and to use more item types. However, Fennessey (1982) found no relationship 
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between training, and pattans of ^ use types of tests admiRStered. 
Knowle^ of testing and measurement principles and ojncqjts was not ndated 

to U» aoKwrn of time tesKiteis allocs to testily or 
tfsty tfiflgh^g gnroanffited themselves. {fowev»-,Ydi (1980) found that trafaiiiig 
in toting ffiKlfl^asiBmeat was «gnirs»nayiefa^ to Uie use, m^^ces^ 
^{noperuse^ofstafHtardizedteiaresalts. But. be fooiKl that tnunii^fai testing and 
meffioienteat (tf the gnxip affveyed was smiewbat limited. Amxntstudy 
(Green & Stqger, 1986^ rqxHied diat tlw nun^ of omoses teadiers 
tidEon in te^if^ ^ n^aniiemm little ieIation«iilp t^ 
use iHtt was significatiiy rdated to the use of cmitanpcnwy meaanemcnt 
jTOCticcs. Th^alsofoundmalcleacliOTtohavcmorepoativeattitudesiowan! 
aU aspects of testing than d^fonales. Fuitbennore, teachers who considered 
their own test results to be of valiK and who viewed tests as goK^Iy rffcctivc 
and fair seemed to use tests mcwe ext^ively. Teacheis in goiend rqmited 
themsdvcs to be ccmfcHtabte with thar knowkdge irating OKi iwe <rf tests. 
However, Green and Stag« (1986-87), and others (Gullickson, 1984; Leiter, 
1976; Newman & StaUings, 1982), pointed out a !«;fc of sophistication, 
''possibly even a lack of cwnpcteiKc, in testing tectmiques, particularly in 
^^i^icaI{Uialysisort{sti^ults''(|i. 53). Newman and Stalling (1982) f(MiiK! 
liule change in ccxnp^ency in mea'iurement asd evaluation of teachers since 
Mayo's (1967) study. 

Conant (1963) in his book entiUed,"The Education of American Teachers", 
pointed out deficiaicics in tl» iMTq»rati<m of teachers and ma(te ^Tecific 
ref«CTce to the impoftaiKe of testing ffljd mcasuremeil ftsr teffi:ters in order to 
properly test ami evaluate tteirstuctents. ThisbotAdrewara^xMiscfirOTithe 
National Council rai Measurement in Education in the frnn of a sponswed 
symposium. "The Imirfications for NCME Policy of Conanfs Book" (Mayo. 
1964). Threse papas deliveared at the symposium were publiidied in tlw first 
vc^un^ of the Journal of Educaticmal McasuremenL 

Mayo's (1964) paper was one of the three delivered at the symposium at the 
NCME annual meeting in Chicago. This paper reported the results of a survey 
of teachers, principals, supmnlendents, college and university profirasOTs, and 
tMting aiKJ research qjecialists. The airvey solkiited the ratings of these 
respcNidents to 70 competemaes derive! fiom the Outline of Needed Competen- 
cies fran the NCME Committee on Prc-serviw Prqraration of Teorlwrs in 
Educational MeasurcmenL A majority (rf the competencies were deemed to be 
imp(xiant by these expats. Only two of the 70 statements describing compe- 
tfflides wrc rated to be "Of Little Iroix»tance" on the average. The rcsL were 
judg^ to be "Desirable*' w "Esseirtial'' by a majOTity of the ttaets. 

Mayo(1964) indicated ihatcompetenciesraledby thejudgescomprisedfour 
cont^t categwies in measuiemrat and evaluatirai; construction and evaluatiai 
of clasaoom tests; standardized tests; uses of measurement and evaluation; and 
statistical concepts. In fact, the textbooks in testing and measurement wWch 
were subsequently written attempted to cover these amtcnl categories and the 
underlying specific wjmpetencies. In fact, tte contemporary measurement and 



airi&sid)seqpientQi^(Mttyc^ 1967)*^ 1986^ p. 347). 

As c^i^« teacb^ do ccmski^ ^£ii^icd 
impOTtant ami th^ put mem &n^fi&^ cm ncm-tesl daia, in thdr evdiMOkHi ctf 
In cmto to ^mnine tte relive pmsp^ves erf te^iJ^ »Dd 

quc^kmnatre suriy erf 24 pofcssore md 360 teachm from eteoCTtery and 
secmKf»y »:hocris. The ie^>ective teadher aid imrfK^ 
s^itedaUAof67tqrfcs finMn^rtnieaaiiCT^ccAimtc^Sarie^ 
lltepnrfessoi^wa^a^kedtoimiic^ 
toi^ in tli^ {ff^s^rvke 

iiKlk^ fte idimive etnpivi^ they bdteve dKxild be i^^:^ on each 
Rtting8(»ittettq»(»wmsiHiunedfc^eKh(XHit^c^ Tten^tscrfa 
mtiltivtfiaie test ^wed significant di^repancies betweCT teaclK^ mi jmics- 
sofs n^can laiings for the eight COTitent HowevCT, on five erf the 

content categories univari^r^ls^i^ Tt^ best agr^nent and the 

higher assisted [^kirily by both gnnifs was on "Hhe i^ep^^^icm of examina- 
ttotts**. TTieotbcrtwocategCHiestbegimi|»apee^ 
scmng tests; and goierai ass^sment infonnation. Tb& categcmes cm which 
professors and tethers disagreed cm the emfdia^s in pre-service ediK^aticsi 
weie: mm-te^ evaluatkm Ktivitk»?; fcxmative evaluation; sumnu^ve evalua- 
tkm; fegal issues; and ^tistkrs. Pmcrfessc^' n^^an ratings were signtfkanlly 
hig!i^ on 1^1^ and lower cm tte oU^ fcmr cat^ori^ than those of tf^ 
teachers'. This is not suij^ng. and Sletz (1979) raggested that 
nK^urementq)edali^ hfKl relatively in^un^peicei^LicHisof fieacha testing 
behavimsandi^eds. Tt^^tuaticmh^mttchang^sim^ethen. Buttheprobtent 
isnotofpocqrtiiHis^wb^herffircttfatec^'ifi^ what is 

rdev&nt ami i^i^c for dasmxmi t^hers to acquire during i»e-savice 
eihic^ion and in what they will a^ire to (fevekq) further compoem^e because 
it is ioipcHiant for their professional ccmipeteiK^ 

Test Use 

Gullickson (1985) rafted tteu elementary te^Im rdy for their stuctent 
evaluations more (HI non-test data than test data* Ontteottorhai^,sec(mdary 
te^ii^ put more empl^s on test than mni-te^ data fa student evali^tioi 
purposes^ He fcMtnd diat both elementary and sec^^ncbry teach^ do not put 
heavy emj^sis cm cOTinwrcially i»eiwed te^; howcv^, s^ondary K^^ters 
put wronger emi^msis on these than eloi^tary teachers. 

In an earlier sbidy, GuUicks(»i (1984b) sought to dctcmiiK^ te^^hei^* 
perspectivesof their instn^^tkmaluseof tests. He reported thatte^fi^i^ believed 
that they were on their own as far as testing for instnK:ttcHiaI purposes was 
COTcemed. FurthcnnOTB, te^h^ "an)ear(ed) to be comfortable in their 
kiK)wfedige ami use of t&$t&, perhaps too comfort^le** (p. 247). While teact^ 
felt they were pressed for time, they preferred to give tests frequently and 
believed that their preference was shared by both idmtnistrators and students. 
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Gfdlid^son also f<Hmd tt^ 

a potemafBact beymd titeg[m^aib^ irie> viz,, assesanCTt, Teachmviewed 
mcae pc^ wly fcH* li^isiBt^ IcwiOT cog^h^ 

McasTOCTam specfalias, Grcmhirel (198 1)» oni^iarize th^ iten ar»ly- 
sis data provkle a basis fcsrusefidcl^ dtscusstcm ctf te^r^ts, (m imed^ 
wOfl^fOTgemalio^TOvai^ofcI^ iiianK:ti(»i, SMi inproved ki 
c(»«an^iQO. However, demc^tt^^seaHidfflyteacl^bdte^ 
cotrs^ in measuranent ami evaluatkm ^khiM efnirf^i^se the <tf test 
I^q^mtioii , tmt ttey do not ccHisk^ ^Mi^ical mwXy^ of test icwUs of mch 
an impcslaiK^ to be r^vant at tte pro-s^ce lev&l (Gullidcscm, 1984a), 
Widmit res^ng to ^ Aicat analysis, testers believe itsk te^ to poss^ 
iKmidpsycfK»netrici»v^peities(I^&G^^ Fiirtb«nicKe,te«^»s^ 
not pocdive item aralysis as i^dca! in die clasaoc^n setting (GulUcksra, 
1984b)* Hm^e.'VitiKmtsystematk^analy^ofUieset^it^K^tt^ 
asstinBicediatd^irtestsfuncttcHiasd^li^. Atbestdiisnmistemrbmrealtzt^ 
less d^ die fuU p(^ntial of didr tests. At mffsu nmny tests may misdirect 
teaclu^ and d^ir stiKfents'' (GuUfckson & Ellwein, 1985, p. 17). 

Practical PimUems wth TeackeuMade Tests 

It stould be clear diat the otots or pnAtems commonly found in tc^her- 
made tests could be g^ralized to instructors at die post-secondary level As 
mendoned earlier, tte {^oMems to be enumorated bebw stem from diree basic 
reasons: dn^ (niessur^; in^iequate sduUs in testing and measurement; and 
inoHtect esdmate of sttt<fcnts* ^ility levek (Miller & Erickswi, 1985), The 
errors diat teachers commonly make in testing s&Klents have been summarized 
by Ebel (1980) and tte foltowing list is derived from diis souice« 

1 . Te»:iKrs tcml to rely prinwily on their own subjective, but presum* 
ably absolute, standards in evaluating £K^evemcnt 

2. Teachers tend to put off test pr^aration to die last minute, then diey 
do it on a ''catch-^-catch-cm^ f^is* 

3. Many te^hersadminister tests dtat are too poorly planmi, too short, 
or too inefiicient in form to sample adcquatdy the intended contcntandabilittes 
in the subject 

4. Teachei^ often put too much emphasis on trivial or unnecessary 
details in dv^ir tests but neglect to include basic jmnciples, understandings, and 
a{q>ticati(ms of the subject* 

5* Tethers often write le3l questions, bodi essay and objective, who^ 
efTecUveness is redm:ed by ambiguity or by irrelevant clues to the ccOTCCt 
answ^. 

6* Many teachers unctercsUmate or overlook die influence of sampling 
erroi^ on test scores. 

7* Most, if not all, teachers fail to examine die effectiveness of d^ir 
tests by even a simple statistical analysis of die items or il^ results of their tests. 



Bei^fes Ike fal^fied ptd^ns df l6fK^Hnd& t^as» ^tt^ 
mray tt>r^mf U^i^^oreastbegr^. A70%%xxm<Hiati^ist^^inm^ 

aie admiiiii^»ed with v^ykig wd^ttts, tbe 

to tte unit of mea^ament mvdml itt esxh mea^Hein^t occas&»i. Ttese 
ffitaatkmsre^tini^^KY^Riatetesdi^ givaitoti^diffi^imt 
secticms c£ the subject being evaluated. 

Expert-Made or ^an^kirdh^ Tests 

Simi&niizedtests«oi»»ltyi»qMBiedbytem^ Atypcalteam 
iiivcd ved in tiie aHi:^nK;tk»i a ^^Kiardia^ 

expeit, »ib^t area ^xcmli^ (s), tc^iiH^ m cc^sultants erf* a »d)ject, 
editesB^ffiidassocmtes, The number and type of experti^repiesCT 
f(^(tevek^Hi$a^im(bidized testva^ This is beewse the 

re^^mices a vailabte and (he coverage in t^uied by tl^ t^ one ncrt the same from 
(e^tote^* 

A standardized test is intend to use ova* a diverse range of ^ility df 
^uitems and enccmipa^^es a variety curricula ffi:jro^ a large numb^ of 
jtffisdfctkHis. Thefefcm, a stambrdized test u^ly cov^ the general and 
ccsnnKm knowledge domain anKBig sev^ curricida. St^h tests aie as 
sens^ve lo difTerent instnK^tkHial c^Httaiit md ixxx?esses ^{rfia^zed (hiring 
instructicHi by a teacl^ as are the tetK:lw-fliade tests* Standwdized tests 
emphasize oidy genc^ skills am! krowledge compcMi^nts which are assmned 
to be expected at the specified age (M^gn^ levels* 

For testing the onnnion and g^i^ krowledge domain a {^staking 
attaiti(»i to (tetail is i^'d in the devek^io^nt ami lidd trialsof a standardized 
te^ Fir^ofallactetaitedtabteof^)ecincatioi^c»*aUue|mntof the^ 
{Ht^wied. FiK-fk^ng this all relevant curricubi arc coi^t^ and 
q)ecificmi(N)sare]niq>areddi^ares^ttoanumb^ 
Secc»id, items are written. ThinI, items are ^kninistered to soiall samples fhmi 
the target pqnitatic^i. Fourth, re^xMises to the items are analyzed and those 
elements of the usable items, are revi^ which have been klentified to be 
ambiguous or ntHi-functional* At this stage, unsalvagealde items are discarded. 
Fff][h,twoormorei»eliminaryf(mnsof thetestareass^bled Sixth, each ctf 
ttese forms are miminister^ to small repre^ntati ve samptes of the pq)ulatic»i. 
Se>^th» results of the^ tthninistrations are analyzed amt instn^ticMis, items, 
item anwgement, ttc. are revise as i^essary. Eighth, eadi of these forms are 
administ^ed to r^Hesentative samples of reasonatrfe size according to a well 
defined validati(»i plan. Ninth, results of validation administrati(Hi are summa- 
rized in a manual. Some tests provide extensive details in a supplementaiy 
manual fcH* experts and for use beycKid the classroom. 

Of course, the (fetalis sketch^ above are for an ideal standardized test. 
Many standardized tests do not meet alt of these requirements. In or^ that test 
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iiialBOTfflidCOTSiiniOTareiisovUtedwfthsoniegBkldH^ 

<m MeasiiimOTt Bi Bttaatkw, tte An^ican Psyc^^ 

the Americra EdttcakMwI Itesearch Assoctolon teve imj^ 

mis for Eduadkmd and Psycbohgica Tests (1985). fa s|Hte ^ th^ 

giiideliMS, ™y iwsOTqndoiB te^ mtos la^^ 

not ms&i even mimmally Owse ^andards. Tteefoie, consumera mf& to 

bewme <rf tl» feci tlutt the shew avaibbilUy irf a te« umler the title of a 

^andantbsed test does not necessarily make it a test that meets the amidards. 

ni^is why die pidjiidied and cooBnerc^y avaibbte standardized te^ aie 

reviewed by experts and these revfews are found in many journals Mid in a 

compichwisive and aith^Hic source, "Buios Mental Measumient Yearbook", 

which is publbhcd about every four years. 

Rom the avaikble may of strndadizied achfevemenl tests it is possible to 
sekja a lest which can s«ve a variety of uses. Among Uwse are: fastnKtional. 

guklance, administrative, and resefflXJh. In tlw case of HislnKiKmal uses, 
standardized tests have be«i found to be effective for tlmevaluaUOT of teanii^ 
outcomes, evaluatiwi of leaching,evalaation(rfciariculum.leanungdiagnoffl 
anddiffenauialassignnientswithinclass. However, standardized tests may not 
be quite effective for graJing md motivmitsial purposes in instmctiraial 
situations (Mehrens & Lehmann, 1987). 

Standardized achievanenl tests can be effccUve for occupaUonal and 
educational guidance. Among the variety of administrative uses of standard- 
ized m:h»vemcnt tests are for setectirai, classification, placement, puWic 
rclaUOTis (informational), and curriculum planning arel evaluation (Mehrens & 

Lehmann, 1987). «. . , . i .• r 

In contrast, the teacher-made tests are most cffecuvc for the evaluation of 
learning outcomes, grading, and motivation. In a study of 445 first-year 
teachers. Hall, ViUeme, and PhiUippy (1985) found Umt tfiese teachers consid- 
ered teacher-made tests most usjful for their sclf-evaluaUon and fcff roouvatmg 
siuctenl learning. These teachers, it seems, attached the greatest weight to state 
assessment results for judging the adequacy of leaching and the adequacy of 
instructional materials. Within each of the three lest types, teacher-made, 
district-wide stantfardized tests, and state-wide minimum competency tests, the 
beginning teachers considwicd test results important for decisions regarding 
students' acadcmfc progress and ^cx decisions about the diagnosis of student 
weaknesses. 

Whereas onuormore forms ofaslandardizcd test can be reused, the teacher- 
made tests arc seldom apprc^riate for reuse as a totality, a few items may be 
reusable. While ic^ber-made tests are sensiUvc to instructional processes as 
well as to the tau^t curriculum domain, standardized tests cover only the 
common knowledge found in several curricula across the target population. 
Teacher-made tests allow inferences of student achieveraait only in terras of 
absolute standards, standardized tests allow inferences relative to other stu- 
dents. 
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If^imstkH^ and te^g ^^iimm^its bxo fHv^iessively becomfaig iik»^ 
cof^l^fflidctidlaigisg. ScfKxib mi bmg^ c^tff to miidmts finmi a mam 
sii^l^cti&mdbKlgrracd. QassiaHiisiiokH^^iiqms^aiimD^ 
idM%,iQ^i&ide^a«iieadiQess levels* W^tbenewcQcrfantolMiisoid^ 
atocdtyffKposav^amlyetbahi^cimic^^ 

txMh in pnces^ md cw^m^ doooamls cm ta^^t^* ing^dty aiKi creativity 

havemcm^d. Stmk^ aie expected, tteefiro* to K<pii^ 

decbmtive knowted^ (eomeat) as we& as pnocedural (tow) and ^m^fe 

(wl^OT{di»miiig&ins)kmiwl&[%e. Thesediai^tomimy maymnsigiiala 

dq^rtiTOfhmi wbtt^lmcdsw^dbiiigciraq^ Howev^, 

withexidkslt^^mimtctfcrt]^ctivescrf$c^^ 

types, tHcssigesfw assessing tticm have 

As anexmii^ d'the inciea^ c<m^^xity df tt^c<»i^p 
Sadctfdiewanisacaseinpc»n(* Foltowinganint^vestiMlycrf'v^rkH»»sect(m 

the Saskatchewan population, of the cumcida in tme, and of professtonal 
ii^Mit, a Minsster's Advisory OHnmiUM 

issued a rqxmenUtled/'DirDcticms^ in 1984, This r^KXt provided a gen^ 
fitmewc^ fcH^ tf^ futiiie Sa^caicl^an cunicd R^whig this rqxHt, a 
CcHe Cinricium Advismy Cteranittee was ai^)dnted by the Minisi^ <rf 
EdtK!ati(m in tte f all of 1 984 to id^itify and re^^ 
the c<se cmtk^ulum. Hiis ccrnmiiuee is»ied its rqxxt, 'Tinogram Policy 
Ptqxisals'*, in January, 1986 which iwovicted a frwnework fc^ K to 12 cunicu- 
limi in tte province- The two m^jor omipcmCTts of this currkrulum, being 
implemented in Saskatd^wan in a planned and systematic way, are the seven 
required areas cfsmdy ami the six comnwn essen^ [earnings. The ^vm 
requited areas study are tte ccmventicm! ^ject area grwf^ Imigu^e Ms; 
mathematics; Kiem^ social i^udies; teahh educioic^; ails edtK^atton; am! 
phys&:al eductticm. Hie comnuHi essential leamti^, a ctesignation not usual in 
tte literature but ^mcwhat ctf an ^ligma m Stsl sigttt, are mMhing more ttum 
gei^ral ^Is, attitmtes, valt^, am! iqpeciatioi^. These are also simile to 
various intelligences identified by Gardner (1983). Specifically, these general 
skills are: communk:ation; numeracy; criticai and creative thinking; technologi- 
cal liter^^; indepenctentieaming;mtdpers(»ialandsocialvalt^andskilts. The 
ccxnmon e^ntial te^ngs are intended to be incorporated into the required 
areas or sub^ts in the school curriculum. 

With this kind of curriculum emi^is, it is ctear that teaclu^ are required 
to test not only the three types of knowledge within each subject but also the 
generic skills, attiuides, am! disposittons. When the testing demamis of these 
ccHitempcHary curricula and the teacher prepar^n curricula of programs 
acix^ tln:^ nation are juxtapc^ed, the envious oonclusbn is that the training 
im>grams are mil in keqnng with the assc^ment im^iency needbd by the 
present and tt« future te^hing force. The first ^uticm is to inclu(te mxrt but 
^[ptq;^te testing ard measurement compcHients in the jHe-service teach^ 
education programs. Only the inclusicHi and mandating oS such curriculum in 



be tao^ (tmkig tto fcMind ^hff^^^ 

Also, it dKHdd be recqgni2s»t tfmt 0^ 
nrativated and sftimul^ (bring ewiy expc^ie to iriewirra^ ami 
evah^tKm^woftiklbecwftiUrogin^ Wei^to 
bol^UiemeasiiimaU8iidevaIiiaik^»p^[^ 

dis&kai and the piavfaKiM ctf 6diK:mk}n hvds. Only througli 

ml va%^ tiBining in tt^ aiea9 hc^ 

to export^ far^^dcAiiig the fotisiede'^^k^ents in ccMnputmzed ^iai^ve 
testing and tte conuneiciai item 

AmMh^ ix^bility is that currfeuhun guidelines fcr varicnis subjects be 
gr^- mi age-specific ami pnovkte fcruse by tesK:hm item t^nks linked to 
v£BicHi& if^tnKdi^ial {mx^e^t^ ^ ccmteitfs sudi as those imduced by the 
lMtnK:tiOTal (»jectivesExdMngeatUCl^ tmd^ t^ 
niis (omsit provictes a omven^ ssx^ thn»igh a sc^isttcated identifkmicHi 
of items defied fcr a measmemrat event I imdmtmid tfmt attemi^s at 
(k^^tefring item bimks in C^tmio have been m^btu I am not quite aware ctf 
theexteniandqualityofusecrfthissource* However, if computciiaedcMr hard- 
cq}y formm item Iwmks arc (fevetoped, Iten a poiodic evaluation of the 
effectively of utilizaticm and sittisfi^^lkni with it ought to be carrted out 
During ti^ develq|»i^ of ^ banks, it is impc^fant that te^l^rs who have 
taught a particular subject f& a ^)ecific gnrie tevel be involved in the (tevelop- 
ment of them, Altoit^vely, curriculum specialists who are in toiM:h with 
tead^rs and have »xess to instriK:tionaI activities of teachers be involved. 

A perak^kHis (Hoblem often identified in tte literature (e.g., Gutlidcson, 
1984) is tte te^hm' l^k of use of even riKiimentary statistical analysis ami 
then* ability to inter{»et tsA lesults and item analysis. This iroMem is inheient 
in tl^ lack erf c^mfidcnce and cxmipctCTce erf many te^K^hers in theff use and 
manipulation of num^cal infcmnaticsi. This is symirtomatic of a corollary 
proWem of innun^racy amrag many of our adults in general (I^los, 1988), 
The mly solution to this probtem is the geiteral impiovemrat of instruction in 
and attituite towanis matf^matics* Unless that h^^pens we will continue to 
witless unwillingness cm theimrtof many undeigraduate and grmluate students 
to bei^fit from and enrol in measurement, evaluation, and statistics classes and 
{Hogmms. If we make tf^ measurement, evaluation, and statii^cs components 
ccHnpulsory, without ensuring that stucfents are {Ht^Iy mcHi^ted to leam and 
withimt making these students realize the impCHtance of the krowledge in this 
aiea for their teaching function, pie-scrvice teach^ may just go through the 
pKes to satisfy the requirement for cenifk^ticxi. Our interaa, to improve 
assesanentF«acticesofte^hers,wouldnolbesavedbythisapprowh. Wehave 
to seek intefesting and innovative ways of tc^hfaig and motivating students in 
this domain* It will be ideal if students seek these compoiK^nts even if these are 
available as elecUves. That is a formidable challenge in the i^ar future. 
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In-service has been used jjimmyiiista^^ 
rq^otolie €^ pacti^ It siKHild be cmly to tffing fixrtfi 

iiewe^ai^ iiuiovativeanHt[^he8 to 

to a pie-s^ce ediK»tion in m aura as knpoitam as evdu^km m whkh 
leach^ spend a cofii^f^ai^ aimnmt 0^ 

InsCTvtee ^^icmkl »q^rfi»i^t and C£mi[4emait iHi&-servk» hsckground fai 
my impat^ teachteg fimotim le^^bers ate expected to perfcmn. A wdll 
ecb^rml sad infomied teacMng foiiv stmld iokk^ in-senrioe in their per- 
cdwd oml aiea wb^ incHns? than the twk: ts^^ 
Ini^f^ by iK> n^is is a substitute for a ftnmal coitfse ]»t^rain in m 
liiq>^ta^kfiowIf^gectoni^« We aie too often luiied into dcnng in-service at 
tones ^ to the clbitts quite in^qm^mate for it to be effective. 

Gmclmkm 

The paper siBveyedtl^ use and eflectiv^^ss of teact^^nude tests. Ithas 
been pointed out that evaluation plays a imminent role in the teaching function 
of a classnx^n teacho^. Teadier-made tests me tte pim^ means of grsdii^ 
indents ^ povkUi^ feedback to the students «id te^:!^ Incite of this 
cnK^tol xt^ of teach^-m^ tests, many te»:ti«^ do nm ^ 
service education in measurement, evaluati(^, and statistics. It is argued that 
education in this luea be mandatcny and effective. 

Te^i^-made tests have been compared and contrasted with the expert- 
made m standardized tests. Finally, a set of proposals for im{»oving teacher 
cc»npetence in measurement and evaluation have been tm6&. 
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Considerations for dbe In^^ien^tation of an Ungraded 
Prinmry Program 

KfUhedne A* MacRury 
OiUario Insduite for Studies m j^ucadon 

In 1989,tl^NftnistiyofEdtK»d<Hifc^tteProvinceof Bri^ 
the deddcm to iniplCTiOTt an ungrafed imnwy program fcH* dtibb^ in their 
flmfoiirye^ctffcOTnalschoc^^ WhileaMW^ 
(kx^tunait ^titled The Primary Program (1989) dM pt^smi m ov«vtw ctf 
ctesnxHn i^^^lkes for te»^^ in die ung^cfed s^ng, the Mini^ did not 
tekox a ^ of policies Umt mmld have guided tl^ implenKmtatioii of the new 
program. Within this ccmtext, the i^nesent i^per (^siders five issi^^s that aie 
related to tli^ tmplmiaitatk») of tli^ tmgra^ primary fm^^ram and that may 
require spectfk^aticm in isovlncial policy gutdelmes. 

Hie five issues for Ihe tmplonentalion of this ungracbd primary program 
are: 

L Documenting the features of the ungraded pnogram at the level of the 
school, 

2. Designing the program within the context (rf teaching resources^ 

3. Develoinng studeitf asi^ssment policies with respect fo monitoring 
progress and diagnosing devdqmiental problems, 

^. Ftoviding eqiml oppcHtunily fcsr each child to progress, and 
S. Facilitating the child's transition frcnn the ungraded program to a gr^fed 
program. 

1: Documenting the program features at the school leveK It cannot be 
assumed that the ungraded program should be implemented with tli^ ^e 
features across schools or school districts, f(x reasons itmt cmitd include (i) the 
p^qptions of the concept of ^'ungr^led inimary program"" i^ld by decision- 
mak^ ftom difP^nt (fi^iricts, (ii) tte i^ial nee(b of childrrai within particu- 
lar scIkx)! catchm^t areas, and (iii) the resources that are aliocmed for school 
or district level implementation. Thus, docum^tatkui of the programs imi^e- 
mented by schools would be essential to providing an accurate descriptim of the 
imovince-wide program artd a meaningful understanding of program elTective- 
ness. 

By necessity, it would at^pear that implementation of a provincial program 
will have features that differ at the local (school) level In the research m the 
eiTectivencssofthcopcncducationmovcmcntof the late I960sandcarly 1970s, 
differences across ^tes w^ seei as problematic. Several studies which 
attempted to synthesize the research on the effectiveness of open edtKi^ation 
programs repeated on Ihe diflicully of ccmiparing results of different studies 
because f the variability of die op^donal definidon of "open educadcm*" with, 
for example, the concq)t of openi^ss emphasized as a physical space in scrnie 
studies and as an educadonal structure in others (e.g., H^witz, 1979; Peteson, 
1979; Marshall, 1981; Giaconia & Hedges. 1982), In fact, Marshall (1981) 
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sugge^ that the coiicq>tofope»iediK^ 

of being more i»)ec^ about the diffi^nmces in the component features erf the 
in^deoi^^ programs (p, 181). 

H^e were also difraeiu:es m the way in whkJi tlm feature erf' open 
^tucat&m im^fBms c^^orized acio^ diffemit ^udies; foe exanqrfe, 
Gia(^HikandHedgc9(l^)ami»fedthe€^^c^ 
Weiss, FirfieraidMiiselb(i972)andWalb^airfThon^ 
Of tteir sevoi cai^cv^ GiiKxuna mkk Hed^ cof^(feied four im^ram 
featmes to be edueittkHml tieatmaits: 0) ^ ^ tte child in leaning, (ii) 
diagnostic evalimicm, (iii) materials to manipulate md (iv) faidivUraihied 
insin»:ti(Hi, md the other three, »faninistiiaive w CHgmitzaticmal fe^uces: (v) 
muhi^ gfou{Hng of idiudoits, (vi) opoi sp^e md (vii) team teeing. 

t^ ungiacted immary iHt^imn in Britidi C^^ 
to what ejOrat the i»oviiK:ial gui^in^ will imvicte tti^ ftexiblUty fcM* school 
principals and teacheraiwed to the im)gim to meet 
or sctKid district* HoweverJtisanticii^tedibmtt^&)cantmtationcrflHY)g^ 
fi^&ir^ at tte school fevel will be necessary to accurately describe th^{»t>gram 
frrai a {^ovincial pec^)ectiive. 

2. Designing the iHOgram within tt^ ccmtext (tf te^hing fe^)un:^. 

The allocation erf te^ii^ xesmnces ft) the ui^^ted i»x^iam may be a 
factcxrmtheext^ttowhichim)gramfeatuiesc^betm|d^^ I^tmiecent 
lepofts ctf personal expt^i^ices with iniwvati^ prinuoy int^mms, tte fonda^ 
mental leci^irce that seems to have (fetomii^ 

the classroom teacher, (sec e.g.. CharieswOTth. 1989; Oberiander, 1989; 
Rotbenbe^, 1$^.) Two (tf tfKise scmroes are qtKKed below because tl^y 
iUu^rate from the iroider*s point view the ^peiKtence of the innovative 
[msgraro on tte training ami ccmmitmem c^tea^hm: 

From his expertence as a fornix teat : icr ami researches in open classrooms, 
Rottenberg wrote thM ^e^ing in an q)en classroom, &ven in the best of 
cinnun^ances, is very demanding, psb^ for more so than in a tr^titional 
classrooflu Effectiveteacbing in an q)enclassro(Hn requires ccmstant 
c(HitintK>us iniK)vatkm, a sensitive system (rf imKiitOTing stiufents' perfcsmance* 
and weU-devek^)ed skills in maintaining order wittout being authiHitarian. 
Maintaining tte energy ccmimitm»it to do alt this well is difTicuU f(»r trains 
axKl experiem^ teachers. It is impossit^ to say what the toll might be on 
te^h^ who are inexperioiced or not weU trained . or who face re^stance from 
administrators, parents, or childrra** (1989, p. 78), 

Charlesworth (1 989), a primary school j^mlist, (kscribed (he drawteck to 
programs of cmtinuous progress andmulttagegnKiptng in temsc^ the requ^^ 
skills of the teachars invdv^: 'IndividuaU^ in^nK^ticm and regular moniun-- 
ingofpxogress are es^ntial to its success. Te»:l^rs in this type of program must 
be skilted diagnosticians and planners^. Many primary q>ecmlistsi, amcHig them 
myself, believe that open edtK^ation faded in this country ncA only because of the 
l»u:k-to-basi(^ movement, but also because teaches were not in on the initial 
plannmg, were not thoroughly trained, and did not receive rmled support (mce 



er|c ^ 



ttey gcA ^ffited ,^ CcKadniKms progress motti^ grmq^ (femamb 
10). 

Fh)in tti^ dtesmiHicms, tewt^ in »icc^ 
mim be ddDed at i^imng, RKmit^^ 

i^IIs ai^ techniqt^ may ^trnnu^ ^ 
U^amcqnctftteungiadedim^r^ The teKh^sns also cfe^l^ as having 
a^MHiglevelofccmimitiii^t^ To be successful 01^ tinie« 

a im^rani may ato need to enmie that the scIkx^ has a ^img lev^ of 
commiin^ tt> tl^ teaclH^ invcrived, {Hovidki^ 

timt may iiK:hxte teact^ assists md omimtaized f^lities to in the 
cffg^icsatii^ of student d^forieccmiing, syntt^zing and repOTling on stucfent 
progrc^ 

3. Developing sbident a^e^neat pc^ci^ 

From the draft 1989 document, evaluaticm of individual audent iMOgrcss is 
to be based on ''multiple (rf^^vatkms (p. 154^ mid "a variety of ote^vation 
techniqi^ to ensuic leliatHlity and reduce the pcmiMlity of human mm in 
judgnK^nt (p. I MOr. Tte document did not di^i^ the vmom imrpc^ an 
assessment can have; si lea^ two ass^ssn^t purpc^ , mcmi tod ng pnogiess and 
diagnosing learning difliculties, might be c<m»(toed by Nfini^ policy. 

Restrictfng the nuHiitc^ng indent jm^^e^ to c4)sarvmi(Mial techniques 
raises i^i^s about the validity and leliaHIity of the inteipretatic^ of indent 
rqH^« It is not known to what extent t^:h^ m^ (tescriptive md/c^ 
evahiative obs^vations with ccsnnKHi ^anctods, aid tte reliability of obscsva- 
tkmai descrifAicms moss tec^rs cmnot be e^imated wittout fuith^ study* 
When the teacl^r and ^ucknts remain constant, it is easkx to assume that 
descriptions ov^" time are n^anlngful because tlK^ has been a common 
refeienc^ group. However, when iN? teacher or iIk? students change, it is ncM 
clear th^ observations ov^ time are reliable. Tlius, the genemlized assessment 
of a child's progress over tinw may be limited by the standards of reporting by 
tte child's different teach^^. 

The impOTtance of early diagiKKts in the firsi years of schooling might be 
iUtt^ted by research m reading in which poc^ perfoimance in the early 
primary gmcks has b^ found to lesKl to poOT performance in lat^ years (see* 
e.g., Hc^ and Packard, 1985; Butler, Marsh, Shef^^ard and SIk^I^* 1985; and 
Juel, 1988). On tte tests of a four-year longiUidinal stud^ of 54 $tu(knts in a 
low SES schod, Jt^l made this fcneboding t^^s^aticms^: 

A vicious cycle seemed evkieiu. Childrai who did not develop good word- 
recognitkm skiO in flr^ gtddo hogm to dislike resHiing and re^ ccmsiden^ly 
less than good readers, both iniuid out of school. Tliey thus lost ti^ avenue to 
de vetop voc^ulary, coiK^epts, ideas, and so on thm is fo^r^ by wide reading. 
This in turn may havecontributed to the steadily wideninggulf between the good 
and poor readkirs in reading ccnnprehension and written stoics/* (Juel, 1988, p, 
445) 



If sycc^ fai leading is vttal &om tto first ye»- of scbooUng, 
dbgnosra (tf imlh^ difficubbs irauld seem t(> be ffii 
suufeata^essmem. If dj^rvational technitjues arc the only axepted meihod 
a{ (&giK!ds» tto m^ly^g a^mni^ is &m emy teatAat wiH have the 
(d)savatkH^dullswilb«4iKhtoiii^diig»xm^(fi^iK)^ The 
(faaffc 1^ (fax»ifflatt^ates thtt the useof ^^daidized iesis is hiiq^^ 
primary chikireo (p. 13); if this policy ^ d^gnosiic pinposes of 

a»raai»flt,itmightt)esiimiartolfarowiagttefaid>yoatwidithebidbi^^ The 
pc^e^id of staiuknlized tests may wanant fattim axi^esaikm by the 
Nfii!d^, p»rtkiilarly if the early diagw^is mid tieairoem of deveh^nnental 
pn^^ons aie cma^ to the diifd's tet^ sac^& and if standard!^ tests can 
a^i^ the te^^ in it^itifyiog pn^^s. 

4. PA)vidiiig equal o]q?omu% fw each child to impress. 

The fouith issue fas comideiaticNi is whetho^ the imgraded program does 
{Kovide all chtldrcn wi'ii the equal q^xMctunity to ptosKsi,. If inequiti^ cm 
nise because of dii^ueiKes in tte teaning iKeds oS indivklual fiudoits as 
qiedai gnmps of students, slK»dd thej^imaiy pn^ram guktehnesaclmowledge 
tiat te^ng needs duniki be met by aitonative teaching pm:tioes? As m 
exam{rfe, (insider r^eaich which found that chiklren from a low SES 
(sodo-ecoKunk status) background b^flt most fhrni teaching |Hm:tices that 
involve direct in^ruction. 

0»e type o( rese^h suniy invdved the re-analysis data fam various 
Follow Through programs from the eady 19*^ in which ^quoffi:hes fo 
teaching economically disadvmt^ed primary children were devekqied. Arc- 
analysis of one siKh program led G»st»i, E^h aiul Gleason (1988) to 
conclude that Dir^t Instnictknt was the ttmt effi»^ve ^i^syhk:!! to teaching 
low-income stunts in kindergarten. CiccelU (1983) munmarized the charac- 
torislics of Direct Instruction in whkh the teacba is described as being ti» center 
of attention — "as dominant leader aad centril mitlKHity, establishes ami 
^(Hces rules for groi^ bdiavknir Op. 425-426)". Thb is in sharp conuast to 
RcMhenbog's deicriptioo of the te^f^r in the open classroom (which might 
i^y to tht; icsfJter m the ungr^d setting as well) — as hcAh a director, 
organizii^ the onvinmment to meet the i^eds of tl» students, and an instructor, 
leading dm;ussiof]s for small am! large grou{K and occasionally teachmg lessons 
to a large group (1989, p. 73-74). 

Direction aiuI structure were also found to be chan^t^stic of effective 
elementary schools in low SES urban areas. Fsx examine, Clark, Lotto ami 
McC^arthy (1980) reviewed 97 studies on excqttional elemeait^ffy urban schods 
in which successful schools had "clearly stated goals and ob^tives" and 
"structured teaming environments" directed toward improvmg suukntachteve- 
n^nt in reading ami mathematics (p. 469); ami Levine (1982) identifled two 
appro^hes in successful ini»r-city efementary schods in Los Angeles and 
Chicago: (t7group-based,n:iastery-leamingreacUnginstructi(H), and (ii)curricu- 
lum alignment One of the iimitatkms to the generalizability of theeffectivei^ss 
research is that outcome measures of success are typically based on the results 
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erf afflidaidized adnevemoit t^s which are 
measuring growth in sdioob whoe the goals have been 
po^My.ea^ to meaaiieuanglhemiiU¥lechcA» test lion 
tf k wrae assumed that the sanw inantttkjoal practi^ 

an groups of imroaty students, lh«i the unpten^^ 

eosi« tiM schod prindpab «id icachOT have the ftewbimy to ina^^ 

certain aspects of the ungraded progiara aid ^ rctmn flie potk^ 

ffiid pactices m are supportive (rf thefr ^nts' teamfag »»eeds. If such 

diffeences in karaing iKseds »e not cona(taPcd »id accomodstfed m the 

ragradedprognim, themcquiitiesnay becomeai^Mi^ 

pla»d hi their first yeffltrfgiaded schooling. 

5. Facilitaling the child's tcoisition to a graded isogram. 

The Ministry crfBAic^'s (fccision in 1989 was to imptemoit a program 
thatwouldbeungia(tedf(ffUiefir8lfouryefflsoffOTnalschoolkig. ttisassinned 
for this paper that, at the end of thor fourth year, most chiUb«i will exit from 
the ungrafcd iHogram and enter a graded program. At this transilira pomt, 
impratonl decisions could be male that would affect the child's f^ sctool 
experience — for exam^e, deckling to what extent the child can hjndte an 
achievement-oriented pogram and assigning the child to a level of graded 

iHogiam. . 

It is anticipated that transition to a graded program may prove to oe 
Mrticulariy stressful for students, their panaits and tOKihers. One determinent 
may be whcthCT the teachers in the primary program wiUassumearolemeasmg 

tl» iransiliMi for students. For example, will teachers begm to admmist» to 
their Year TTute and Four students the kinds of tests they will encounter m the 
gradcdYcarFivc(Grade4)7 One dUemma is that, if teachOTdofamiliarize their 

students for what lies ahead, they are incorporating a grwied dimension to the 
ungraded program. Another stessfKtor might be the measures that are used to 
assessthechiki'sreadinessfof the graded program. At one extreme, each child 
could be assessed on hisw her individual merits, by interpreung the teachers 
observational accounts collected in the slucfcnfs portfolio ov^ the four year 
period; at the other extreme, student as^ssmcnt could be based on standardized 
test results if a test battery such as the BriUsh Columbia Achievement Tests for 
Grade 4 were administered to all students in their fourth year of the ungraded 

^"^By^Umethefirst cohort has completed the firslfourycarsof the ungraded 

primary program, it is c(mceivrf>le that another ungraded program will be in 
placefcs-yearsfiveandiq). Even so.it is Ukely that ihiscohort, with each student 

proceeding at his or her own pace for four, six or eight years, will eventually 
encounter a graded program in which each student will be assessed relative to 
the rest of the cohortor according to nonns set by the behaviour or peribrmance 
of students outsicte the cohort. It is not evident how well-prepared this cohort 
will be to survive in a competitive learning environment. 
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Candusion 

It is attested tbat five issues be coEisktexed in de>^qnng policies on the 
imgr^ed pdmaiy jprogram duim^KHit Britidi CcrimU^ 

1. Doouaae^ing die jmigisn fetttmes ^ tte level ctf d» scIh^ 
School-level fefiiHcs me likdy to v»y because ctf (Uv^ty in kx»l poc^ 

tions, :^ud^t needs, end avaiMde resources. Thus, accurate desmpUons erf 
programs si this levd inqrf^i^Matkm ste necessary in order lo have an 
ovesBll p^^^tive m the jsoviiK^i^ tm^rem, (^ennhie fe^ures that are 
ccomKHi ami idiosyncratic »:n»s sdio^ Ratify ^ exf^aKiimy vari- 
lUesinanassesann^ntofiai^rameirectivene^ Tol^ttiepnmaiyprv^ram 
in tenns such as *iingiated' as q^wsed to *giafed' OT *devel^iwnal' 
vami 'tr^tioml' is to ovolodic ll» ccm^^ f^ar» tfiat ccHUritstte to the 
iinkp» flni^i«ntatic»» of the psoviiK^ial {Hv^i^ ^ the local fevd. 

2. Designing the piognm within the c(»tfBxt of teachhig lescHiices, 
Pfsm recent mx»Nints ffiitlrars with pn»mal exped^Ke in hmov^ve 

priniffiy progiws, die succe^ these programs (tepen(toit iqxHi t^ 
and c(»nmitment ti[ the te«:hers wl» se invcdved. At the schooi level, 
c(^isdaBti«i of the teachers' skills and oiqxHtunities for professional skill 
dewh^mientinightte^toanM»efeffiiMeini{toaitati(»i Irian. Also,teadhm 
^UMild not be expected K) provide all (rf (he comniitment; the school snd s^ 
district should be equally ctHumiled to their tethers in terms of {Hoviding 
as^stance sw;b as comimterized rescHuc^ to assist with mcmitcmng and 
rqx»ting Au^t progress. 

3. Devek^uig stucfent assessment pcriicies. 

Re^udingte«^8$9esanentprK(i{^theMinistry's(»riginal 1989 docu- 
ment emfdiasized the use of a varid^y of observati(»al techniques and de- 
notnced die use crfstandardiz^d achievenwni le^. Mining frtMU the document 
was a discussi{Hi (rf the rmtge of purposes (rf ass^sn^t and the reiation^ip (rf 
i»sessment to te«:hing in»;tk»s in tte ungraded pn^ram. Two in^puip(»es 
of assessment, oKmitoriiis stuckntpot^ss and diagnosing teaming difficulties, 
need to be d^xK^ed. 

4. Providing equal opportunity for each child to iMOgress, 

It is not clear how the ungracted program can provide equitable learning 
experk^m^ for every child if, as research has indicted, Kime children may 
bwiefit from fiaving nnirc structural instruction than other children, Acknowl- 
e(%ement and supptHt of irdividual w group difference would require a very 
flexiWeimptementationpoIicy. Unfatunately. the potential negative effects of 
the ungrsled program on children who we believed to benefit from increased 
stnKtine will not be kmnvn until afto* (tmc effecu have develq)ed. 

5. FEKilitating the child's transition to a gradol {Hogram. 

The transiticm from m ungr^d to a grated fm^ram may be the point at 
which important decisions are mai& regarding the chikl's future direction in 
school. Two issues related to transilicHi are: (0 wbcthra^ features of the gruk^ 
program will gradually be introduced in the years prior to the chfld's exit from 
the ungraded program and (ii) determining the method(s) of assessment to be 



ERIC 



56 



iml to (te^^nmi^ a chiU's R^ii^ 

Tb^ five i^ii^ fcHT omid^atkm raise tl^se tetosdei: qi^tions about the 
impksiientaticii (tf tl^ tnigraded primly (m^sram in IMti^ Columbia: 

1 . To wh^ wiU sclm^ and s;tKxrf dist^^ 
tte im>vim;ial ungmded {ffim^ 

Z Does tte Nfiiustry EdiB:^ioa ^^0^ 
in impfen^ted[m%ram$ tt> meet the ^pecia^ needs tji indivtdual sttK^ts? 

3. Wlmt mea^ues can ik)w be put in to m^tx the impkmentation of 
the pimary pfc^mm? 

4» What BTQ the it^ssaiy compc^nents of a lesearch program that would 
deveiq) valid, reliable and feasible methods f(xr the assessment ctf individual 
students? 
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What ShouU a Classfoom Testing Program Look Uke? 
The Functionat Factors of an As^ssment Program in Primary 

Classrooms 

John O. Anderson and Dan G. Bachor 
University of Victoria 

Thei^ikmtnamiKtf^^gmmediKrdt^^ lib^tttuc is based 

(m stamlmdized p^)er-p^l tests (Stiggins, Coiddin & Bridg&fcml 1986). 
Chssnxmi as^ssn^t procedures are crften viewed as anal<^ties of ^andard- 
ized tests in thm the asses^oit procedures ti^ by leacl^ 
single instamxs of U^t ^inistnuii^, di^wte fn^n oth^ kinds of classnxm 
activities. Hie manner in whidi classrocmi as^s^ent is to be rompleted has 
bcenthefocusofdtebateforaconsklembtepmodof iime(Bacb^ 1979a, 
1990), FcH^ ex^ple, niea^iren^t ^)ecialists have stniggled with the nora 
referenced and crit^cm-refiaemjed int^iwetive ftamewc^ (Bbd, 1978; 
Pic>pham,1978),standaidsforte^g(EducatkmalTea^^ 1987). how 

to modify infonnatk)n coUecticMi prcicedures to c^mire the potratial to learn 
(Bachor. 1979h; Feurstein, Rand & Hoffman, 1979), ihe use of technology in 
data caiHure (CoUnim & McLeod, 1983), and ti^ freqiK^ncy ami scope of 
measurement (Fuchs, Eteno & Mirkin, 1984). However, the suitability of this 
paradigm itself merits attention* 

Teachers have to measure the extent to which insuiictionai intent has been 
achieved by c«A student in the cla^ This task is ccMnpfex particularly in the 
dynamic envirramcnt of the classnxmi (And^scm, 1990). CcMisidoDed in this 
paix^r are two broad i^ii^: what (toes classroom a^essment lode like and what 
should it be? To addre^ the first issue we def^cribe some findings of a study in 
progress investigating the cla^room a^ssment practices in primaiy class- 
rooms in two schools. The second issue is addressed more speculatively by 
offering reflectkms on what ctmracteristics an ass^smrat program stould have 
given the context of ihe primary clasmwms previously described. 

Classroom Assessment in the Primary School 

The classrooms considmd in this papc€ were in elementary schools which 
aie beginning tc^ implement what is known in British Columbia as the Primary 
Program. The Primary Program document (Ministry of Education, 1989) is a 
draft description of the ediK:ational program to be implemented in the first four 
years of public schooling in British Cc^umbia This initiative was developed in 
respond to The Report of the Royal Commission Education (Sullivan, 
1988). The B.C. Ministry of Education has expressed a (tefinite view on the 
primary classroom and the assessment pr^tices to be implemented within it: 

We envision that classrcxjm assessment will be continuous, formative, and 
reflective of wfmt children can do, and that assessment techniques will minw 



(me sefsnl^ {roce^ (B,C. N^i^ of Echtt^mi, 

In ^diticHi, Uie^[rfK^ crftlwPrini^ P)^^ 
aie^:mt^I^:ti»l,idi^kal,aeAbetic,soc^mKle^^ Each 
^Ktent b to be evalimted in lelaticm toeadi (tf these gfxis. 

The guktetiiK^ fc»r tl^ imj^n^mmkm (tf the i^w Primary Prc^mm m 
Brit^ Cdunibia :»jKX>h, im>vkte m 
be Sieved in cla^rocmis ex^pies of how this migte be 
data collecticm i»x)c^uies (mm^ckmd reomb* mxk smfks, aiKi observation 
guu^), Imfdicit in \hcs& guidelii^ is a call tt^ inqdanentatkm oi a 
substantia], complex as^s^cnt im)gram individimily ciafk^ for each {^mary 
classtx>m in the province ittulfcM^eadistui^ in tte Kowever^the 
programmatic asp^ of ass^sment ^ not ^fches^, nune speciQcallyt the 
following question is not answ(»ed: how (k> the vark)i» compcments qperaticm- 
ally fit tog^her to result in a fimcti£»ial assessment |HOgiam of data collectkm« 
collation, int^retation and reporting? 

Purposes and Procedures 

In 1989, we initiated a study into tte ctas»ro(mt assessment i^tices of 
teachers who were currently implonenting the new Primary Program, Thew 
teachers worked in schools that had been (tesignmed "lead schools,** as they were 
among the first in the province to introduce the PHmary Ph>gram on a voluntary 
basis. Thus, we a{q>ioached the staffs of these two schools with the request fc^ 
eight experienced teachers who would be willing to collabc^te with us in 
examining their cunent as^^ment practices. Hieseimmsuy teachers agreed to 
cooperate with w in describing their assessment i^tices concc^Aualty and 
cq>eratiomiI!y, allowing their classes to be observed,and critically levtewing the 
descriptions and genciBlizations developed by the auth(^. Hie intent was to 
describe practices thitf are typical of those used in the assesimient of stu<fent 
attainment. In addition, four teacti^rs fnnn a third '"lead school** have been 
interviewed to further verify the generality of the {mx^^ures cAserved and 
described in the first two schools. The three ^dlwls are located in two school 
districts on Vancouver Island, The four participating te^N^rs from cm^h school 
had pamllel teaching assignments: one regular teacher of eariy primary pro- 
gram, one regular teacher of older primary children, one m usic teacher, and one 
learning assistance teacher. 

Although this study is still in progress, tte opoational characteristics and 
procedures of student assessment in the classrooms we are investigating can be 
described. 

What Classroom Assessment Looks Like In the Primary Program 

The assessment of student progress is a char^t^stic element of theprimary 
classroom. Assessment is essentially cm-going ^ of the school ^^tivity. 
Although the assessment activities of the teiK:l^r are not hidden from view in the 
classroom* the students are not made particularly conscious of asi^ssment as a 



The only aic€|)ftkHis itfe in tte case iM 

my be t^i^ thtt ioclwk discneie teising m assessmm such as {^p^^ncil 
te^ of varbus types. 

Tte fenefal qsoatkHud cbanK^eri^tcs ctf cla^xxHn assessment ctf the 
pdm^ schools kivolved in this study aie ti^^ed below: 

Assessment is a manual compcHirat of cIa^:orat activities, it is nm imially 
a dkoi^ ac tivUy - k is not sqw;^ fnm otter classnxMn activities, nor is it a 
CH^*timeevCTL 

A^ssmrat can be discrete when tte {wpose is to detennine ccmfinn that 
a child needs mkliticmal a^^ance speckt pn^iBmroifig* 

Crnistantinfonnatimi flow - the te^teriscontiniially (xrflecting infc^rm^cm 
cm ^iKlent perf(Kinance tbM can be leUued to There is 

sonie decr^se in the anoK>imt c^data ^Oh^ng ^bseqi^nt to 
imed* 

Group testing h not pan of tte assesanent proceduie in ctassnxms. 
Asse^nentpnoceduresaie basedon individual studratd^s^^c^ and product 
evaluation. 

Assessment is omducted in a wide variety of educaticmal goal aneas: 
intellectual, physical, social, ^oficmal and aesttetic developmem. 

Assemient ^vities aie targeted m individual students ^d on particular 
a^3ects of the in^nictional (Hogiam (ta^ and pedi^^ 
(tf ti^ stui^t). The in^ructKHial t^ets migitt hz ccmceived of as the path(s) 
stu^nls axe to fdlow in aUainment of the goals. This is nK^e familiffi- fcs* 
sKractemic areas (generally related to the goal qS intellmual ^ velqm^t), and 
» the acad^k: targets for assessinmpurpo^ar&b^ler articulated Although 
a^essment requires that targets be identified and student status in rel^kHi to 
tl^se taigets be rqx>ited, in tte vkw of teacteis, this is not to suggest that there 
am discrete eiK!-poiats for a stuctent to achieve and ihsn terminate further 
learning and ctevelopm^t Th^ is an implicit open-endedness to learning so 
the developmentof tt^ child isopen-oided in the sense of continual, unbounded 
progress* 

The purposes of assessm^t are twofold: to direct instruction and to provide 
solid information to report on sutde^t progress* 

Assessmm is used to identify otulim in relaticm to two a^^ts of the 
classroom: i) the perf<Hmance of indtvklual students; and ii) the goals of 
education. In the first case, students are identified who are either exceeding or 
not meeting one ot more educational g(^. Howevo', this identification is not 
simply whether or ml a child is Sieving goals, but al^ whether tb^ child is 
pcrfcmning up to ability, whether he or she is expcsiding approjK^iate effort, and 
the extent to which pn^ss is t^ing made* In tl^ second case, teachers examine 
the general progress made by all students in meeting various ediK:ational goals 
to determine whether aU the gc^ls are being given reasonable and fair coverage. 

Assessment is multi-modal in that i^hievement can be (temonstmted in a 
numbex of different ways. Fcx example, for the same goal area cHie student may 



cfKX>se to write a i^KA whl^ is B lelevatt dfflK^^ 
ffiKrther tfitd^ nmy with tte iitfc^^ 

a i^y visi^ ait Afl hkx^ coiAl be (xm^ictod ^luivatent, or st tedst 
stiflH^ibnt, fof tte deiiKmAxMrai aOiis^ 

The p«x:edures used genmtedescxiptiw 
imtme. TTw inf wHiaticm is g^faw^ 

by ti^teiK^as^isrecc»dbigttec^c3iltt^^ Insi^c^tbisrahK^ionttei^ 
aie v(^iim^ (rf* tiif(miialk»i (Elected fa radi stiutort. 
Proradures: Inf(»ni^i(Hi colkx^tkm 

1. C»»c»rvaticN] is fmUMAy dK> mist wiitely and finequently emi^qred 
[tfocediire fcnr the oritecticMi (tf infc^matkm for AiKtetit a^esai^nt The 
d>$matiai gei^rally occurs as the stilts are wc»1dfig m prq^cts, i^sign- 
mentscMTOth^scbooI-reli^ta^ Tt^fociisctfcA^eiiraticmistqxNiem 
skills, ^ilities, wc^rk habits, emc^icsial (kvelopnoit, md ^^nq»riate social 
im«ractiOT(coopeiati(»ijOTexaniirie)- TbeuscofcAsav^icMiiotlieasse^- 
ment of stiK^t achievement assumes an internalist knowledge of age/grade 
expectations by the tem:her. 

Two difTenmt types of obserraticm are conducted, in part, varying as a 
function ofageofttechildrai being instructed. First, tea^f^ol^wcactivitira 
as they are displayed by varicHis children in chis^ The t^gets of observation are 
the ongoing, normal cla^rocm activities the sti^ents. Thus, a lecoid of 
emerging skills, ^iUties, behaviour, pattons of social intcmtton, et cetera is 
obtained Recognizing that an cxarflpicof**new"belavk)ur has occurred s^ 
as a prompt fc^ tte sccomi type ofc^^s^vation* Seccmd, teachers select a single 
chiki group of children i^k)r to tf^ begiiming of a cla^ to kx^ ^leciftc 
patterns. In this ca», the observatkni tedmiqi^ are regularized. Targeted 
children are observed about twkQ a day for a period of ai^ximately 15 
minutes, so that ei^h c^ld in the class is ob^rved within a two day span* 
TefiKjhers with younger chiMren (^proximatdy ages 5 to 6) tend to use both 
procedures, tiKsc instriK^ting crfdar children (afqwoximatcly ages 7 to 9) tend to 
use only the latter procedure. 

The main form of obscaraticsi is pmodk, tmg^ed obser>^on of a chiM 
during regular classroom fiCtlvities and rxording salient features of the situa- 
tion. Observation isconducted using oneof two ma^p^jectives: First,itis 
used to monitor students as they carry out daily classroom activiti^ associated 
with the primary goals. Tli© purpo^ arc to note tfej; emergence of a first 
occurrence of a behaviour relevant to ediK:ationat g(^, something tfiat a child 
has just displayed in class (may be able lo do) and to iK>te what a child is able 
to do (can do). Sec(»id, specific children may be targeted when there is some 
doubt that a goal has been met and where it is quite evident a child is having 
difficulty (can*t do). 

Another form of observation could be termed **stnK:tured" observation - the 
observation of the classroom elements tli^ student intmK:ts with and the static 
features of the classroom which provt(fe context for interpreting observation of 
student performance. Examples of the targets of this type of oteervation include 



flie pe^ sttite^ difxises fD wcrtc with, the Ux^ 
re^&gGf tei^em^ii^atc^^tteffiradrthectes^^ 
prelim {o work or 10 play. T(» k^SmmiMkm c^teote^l m 
cmvsxt for thQ iiheipreMioo of othe^asses^ent data 

Z CWtectkmcrfwoik sample: appioxte^ 
placed in pcHtfi^ iiNlivicto 
rec(mb.TtewcHfcsarop^ am evaluated by 
(int^^kms) <rf mrtniokML 

The wi»fc saini^ is pie(kmiiimUy written 
on ammals or jcHinial pages; althou^ art wodc^ {^lems, audio t^>es dT 
le^lkig, and video l^)es ctf ch8t»^t»i^c l)ehavk)^ 
The wi^ can vary c»ie ehild to the i^xt in the soise that fcH^ 
as reporting on a story v^i). cHte child may cmiplete a written repcxt ^Mhereas 
anoil^ may generate a visual (art) dis{day. 

3* O^ckli^'^essentiaUyaliiab^of ttK^p^onnanc^ 
toaninteiKtedwtcon^tiiatachildisabfetodo. Used to nc^ the pie^m^ c»r 
absoice of skill deveioiHi^L Checklists tend to be used primarily by regular 
classroc^nte^^hers. Ini^ysica!educati(m,acheckM(^lrfiy$tcalskilIsi$use^ 
to describe the skills a iftucfent is abte to (k> - these include both physical and 
i»x:ial skills, fot example {»tt<mis of movement Checklists are also used in 
other areas siK^h as noting the ^curacy of rhythmic ctaj^ng patterns w the 
ability seqt^ndng m;tivitie& They are u^ to assist in the monitoring of attitude 
ami eSfwt expemted by stt^ients. 

4. Interviewing: addng s^wlents questions m an individual basis on tc^ics 
of relevam;^ to teaching - e*g.: what bodes a student choo^ and why, in wder 
to estimate ability and intefe^ of tt^stuctent Pnmc^someper^tiveonthe 
student's view of activities and pn^ress and also provides a fociB of the 
process a student uses in ^rcomfdishing tasks. 

5. Studmt self-evaluation: This can take a variety of forms. Once or twice 
a nKMith stu<tents are rcqiKJ^ to provide (write or ^ate during an interview) a 
description of their own strengths, their exp^enccs and ability of woricing with 
other students. These sure retained in the individual student folders* Student self- 
evaluation can also include ^ucfents declaring wten they are ready to read a 
partfcular text cm^ students verbally estimating {Ht>gress in a particular area. This 
is an area of considerable interest to a number df paiticif^ting teachers. They 
want to inomote the use of i»^-evaluation by students to the extant that it 
becomes the main form dTstiKient evaluation. Thepocedures would inclu^ the 
repenting of the stucfent^s own prepress (and presum^ly l£K:k of progress) to the 
parents. TTie processes of high-cmte' cc^itive processing involved in self- 
evaluaticm are viewed as being high irnOTty outccwnes related to the goal of 
intellectual development. 

6. Editing common writing samples: each stuctent is given a sample of a 
student's writing, and then each student in the class is asked to read and edit the 
WOTk. A copy of each student's editing is retained about one piece each month. 
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ProMdureai lofmmatkm reductkm or se^Jt^ to ykU reports* 



Hie leducdcHi ml ktteipn^tkm iofbimatkm ocdtected is ccmcbcted in 
<mkf to (fevekqi iqxms cm 6«A stu^ Ite r^KMs me written 

ddlls ^xoo^^ded by ccHiin^ 

ectoc^ion said progress towarcb ttese goab. The iqions do not contain 
letteigmdes hot comparative ctescriptioiis in ite nonnativc sen«. Tlieiepc»ts 
aiee^ntiaIlyde$cripdc»i$ofrqpiie^taUvectsnt^ A^mient 
inframation is setected that has <^^med m bdng Qrpical of a ycmng- 
ster*s activities (im)ce^ and imxlucts) fm my iqxHtin^ period The 
infonnatim is derived from the intplement^on of the pn)ces5es described 
above. 

The amount of infcumaticm (x>liected is c(msklmible« so, setection am! 
leductiont^totakepii^* The gralsc^ the Primacy Fft^ramaieus^ to guid^ 
tte K^tecticm of infcmnation, Adiild'sf^rpgiessw^ine^of thegralsisthe 
main focus. I^^riptkmsctf^vek^entwi^inafsraiaieaaresn^^ 
information collected through tte proc^scs de^bed atK)ve* I%r example, 
written work samples will be selected frc»n journals that xeffectchwges in skills 
or abilities, stories or other wriu^ {HXKhicts in which a child demonstrated a 
change from invented to conventicHial spelling* 

The stmtent abilities and ^Is are d^cribed in terms of what the child can 
do. The term "can do" is used frequently in tte Primary Rrogram and by those 
involved with it. This emphasis on ''can do"" can also lead to some problems 
when tlu^ is some cofKcm on the part of the teach^ over the child's 
performaiK^e in some are^ to which attenticm should be pEtid (to an extmU '"the 
cannot do*s**). To an extent* re^joiting of prolrfem areas ^jpears to go SOTicwhat 
against*the-gnrin in regard to the ethic of tte Primary Ftogram. 

The writing of the reports revealed a diffk:ulty in condensing all the 
information available on each child. Participating tefchers pointi^ out that one 
has to be awaie that it is *^oticMble'* events that are getting d^CTved aiKi 
generally these events will involve cHity certain stodmts, so« conscious attention 
must be paid to the "quiet kids'* in order to coltect reppesentalive ^mples of 
information. This holds not only for student coverage, but also for goals - some 
goals involve more noticeable, accessible and unckrstandable suid^t perform- 
ance (for example, student performam:e in reading can be viewed as more 
accessible than that r&Ialcd to emotional development). Furthe^^, in keeping with 
the emphasis on describing what the sludei^^t can do, the cements could not be 
simply a non-committal ^'satisfactory** or "mrt up to expectation**, but ratha 
more descriptive of student status supported by collected inf(mnation« 

Questions regarding practice observed 

Our lei^arch is still in progress and many questions remain unanswered 
(more are yet to be asked and many mort yet to be conceived). Many of the 
questions are directly related to fundamental a^>ects of the assessn^ent of 
student achievement We are currently pursuing these questions with our 
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leachea- coIlafatHators bul have not yA estaMished suflid«it resolution to rep(Kt 

Before pie^nlin^ aa spcculaiions on the de&iraWe traits of a cbssoom 

a8sesBniratpn«im,asamirfiiigoFsmi6(rf^tf»gfl{»m 

{Hovkle some useful cont^ for considering these traits. Some of these 

a. How are the iiistm(^ionaleteinenis(tte goals, teaming ouico^ 
tfl*8, objective indkalMs fflid the like) idenafied? How ttoes the teacher know 
whm to focus oa in dMsrvii^ i^udents? 

b. When ccHiducting «*servations, how does the teacher determine 

which students to (^»erve7 

c. Wh^ coiK^otis exist regmding tte sampling of infafm^fon and 
siwfenis? Is the same kiiwi inCcMmadon cdlected <m each student f(^each 
goal? 

d. How raiKdh infwmatitsi is gowrated doi cdteclcd foreffi:h student? 

e. To what extant are ^laJent bdiavkiurs comi^rable in the sense that 
the stwtents could be resixniding to or inier^^g with differeait components of 
tteir educational environmcm yet the teacher c»uld be viewing the behaviour as 
of "the same sxsC. In actuality one stud^ c«ild be exhibiting reading-type 
bdiaviour,anoth€*stoideni"a>ciaI*'behavk>ur,and the teacher could l» viewing 

both stuckaits as di^Iaying aesthetic behaviour. 

f. How representative of tte child's repertoire are the performances 
cdtected? 

TIw above questions are fundamental to assessment of achievement of 
educational goals regardless of the iswedures used to collect, collate, interpret 
and report assessment infonnaUon. The characteristics presented below are, to 
an extent, the answers we hope to find to thcK questiwis, 

Asse^entPrograms in Primary Classrooms: What Should be Considered. 

As«ssment in the primary classroom, as described above, is continuous, 
individual and integrated into classroom a:tivities. Tlw pnx^ures scItkMn take 
the fom of responsive, group testing, nor (to they yield data in a form that is 
f itei- K manipul^ by humans or machines. The assessment program of 
r.'riij J , ' classrooms should have the following characteristics: 

« Hiievant 

-V. ss^ssment program should be relevant in that the assessment itself 
$ V„.d have a purpose that se^cs the attainment of the goals of schooling. The 
- -ocedurcsu«5;Jsi.:\5. t )cdircciIyrelatedtoinstructionalaciiviiics-essentially 
. ai L termed by sc v: as authentic measurement (McLean, 1990). 

2. Fair 

The assessment slrould insure that every student has a good opportunity to 
display target kihaviours which are indicative of goal attainment Contextual 
cOTstrainis such as test anxiety, punitive overtones, and unpreparedness should 
be eliminated. To the extent possible, students should sec the assessment 
procedures as fair. 

3. Accurate 

The assessment should result in accurate information, in that the data are 
derived from representative sampling of the indicators of the goals of education 



being assessed, ai^ of the bdmvkmrs The 

tmlkratcHS ffle to be »>M 

11^ i^x^tuig ^iKmtd be l^:^ on inteiin^^ 
inf(Hmatu»] b^. Ttie inforn}^^ 

availi^ mui inleqni^^e to stakc^kters - tfie (mients, the stfmlems (to tte 
exlc»it fea^ble) ami teachers. 

4. FeaaWe 

The piocedurcs have to be feasible within tte ccxit«t of the [wlniaiy 
ctasmxmi. w tole set of {Kocedtnes must be viewed as **(kHible** by users, 
and must maintain the chan^teitaks of relevance, fainter, and m:curacy * 

5. Syst^TuU^ 

The procedures should be used systematically in the sense that each student 
is evaluated the same basis as any other ^udent In oth^ v^tnds, gi^^ tt^ 
same unck^Iy tng tevel(s) of gral attainment, tte same kinds of evaluation will 
bepnxluced. 

There rfiould be consistency of procedures for infc^rmatim coUecticMi, 
int^pretation and reporting from one ^ucational area to another; and frwn one 
stwtent to another. 

E^h area of schooling should be evalua^ with similar standards of 
information coUcction and inteq}retation« For example, the quality and consist- 
ency of mformation coUectel to evaluate reading attainment shcmld be similar 
to the quality and COT^stcncy of the infwmatioa cdlected to evaluate attain- 
ment in mathematics, or the goal area of lAysical development. 

6. Condensable 

The information cdlcclcd has lo be of a nature that procedures are available 
to reduce it to a communicable size without a significant reduction or change in 
meaning. 

Summary 

A paradigm of classroom assessment based on standardized t^ts docs not 
fit the classrooms involve in this study* Tte infOTnation collection caiducted 
by the primary teiK^hers particif^ting in this study was not a classroom activity 
discrete from others. The information collected was not ba^ on student 
responses to test items but rather to more glotel elements of the cdiKatiOTal 
envircmmenl such as tesKiher requests for iwticular task completion, general 
social interaction or sUKlcnt activity directed towards a |Kirticular student 
initialed project. To an extent^ this lack of fit is unforturmte since tfiere are many 
excellent dma analysis procedures available fw those assessment data that do fit, 
and there is a dearth of systematic procedtues for the classrocsn assessment we 
observed. 

One i^th to revive this iiKompatibility is to change what is going on in the 
classrooms - make the data fit This is not an acceptable route given the 
philosophy of the Primary Program. Another {Mtth is to disregard the inconsist- 
encies and apply the data analysis procedures available. Another is to disregard 
the need for systematic data analysis procedures and assume that individual 
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iniMirvitew. 

A imth we view ^ moA ap|Mq[ma&e is to fuither our imdtea^Amding o£ 
cia^txnn asse^tii^ {m)cediires an^ Th&np 

oimlytic bose fear dasaocmi ass^^ment fmctice. 
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CkssrocMn Ass^sment: Wtmt Re^nJb Pracdtioiiers 

Need? 

I F M Research and EvaluaticHi Inc. 



Abstract 

The pwposedfdiis paper ^ras to pittoittte views of field-bas^ educates^ 
inresfK^^tothequestkm; Whatie^aidicioFiiacUlk^^i^? Cbmrn^us 
gathered finom a number of les^l^ and Strict adminisdatcHs in British 
Colombia tfmxighasmesofinfocmalimervkws. The lesultsweie summarized 
within three c^(»fies: (1) need f<^ better liidcs b^;ween lesearcters and 
pi^tttioi^s; (2) i^od to huld on po^tive attitudes towards clas^oom 
as^ssment; (3) need to <tevelq) new evalu^knitechiikiues to match die m$y 
cimcolum changes anttcipated in B.C AltlKKigh the focus was on cla^oom 
assessment, discu^icHi included i^sm leMed to curriculum and instnictimal 
{imrticesL 

Oossroom Assessment: 

What research <k) petitioners need? The apparent disparity between 
research on educaticHial measurement and classnxm assessment activities has 
received attention in the litc^urc (Amter^n, 1989; Bateson, 1990), The 
national study by McLean (1985) coiKUtded that teachere acquired their skills 
from other practkfoners mi^h as a craft is learned through experience. Recent 
research by Wilson, Rees & Connock (1989) found that teachers beyond the 
primary levels use evaluation to generate marks for grading purposes but seldom 
to monitOT student progress or their own instructional strategies. 

There is clearly a need to provide opportunities for festers to gain 
knowledge abotrt student evaluMion(Sliggins, I988)^ This is impwlant not only 
to imjrave the quality of classroom assessmeitt but also to ensure appropriate 
interpretation of {est results. In light of external monitoring of student progress 
at provincial and district levelSt teacters should be belter infOTmed about the 
way tests arc developed and used (Rogcrs,1990). Because of the importance 
given to marks and gr^ies, teachers need to have confidence in their own 
judgements about student perfOTnanceand be able tocommunicate clearly with 
stucfents and i^rents (Gorman, 1989). 

At tte same time, there has been a call for impt>v^ assessment techniques 
beyond the item formats traditiOTally used for large-scalu testing (McLean, 
1990; SUggins, 1990). Work done in the United Kingdom (Black ADockiell, 
1980) and elsewhere describes new ^proaches to evaluation. 

In order to find out the views of field-l^i^ educators on such issu^, the 
question: What res^rch do practitioners need? was posed to a number of 
teachers and people who work with teachers in the British Columbia school 
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sys^. Tl^pe(^ were ^ectod because of tl^dh^ 

^nictirod ui to allow i<teas to 

(>i tl^ wfK^ imctitbii^ fed v«y ici^ 
eduottional to{^ in geii^ and tcxmAi spcdTK to asKSStnmL Wtteie^ 
research bdates i^miciiter sectkm of edi^atkm for m¥efiigatk}fu teadiing 
deals with tte continuous dynamk^ (rf^ sdKX)!m^ 
tte parts as sqxaiBtefipraittie whole* Tead^seeaiealneedlotidc^tbegs^ 
between reseaidi pa^ake^ e^scm&y in view erf the swecfmig rhanges 
ecmtmned in tl^ &iti^ Cdtmibia Mini^ erf Education ctocumoit titted: Year 
2000: A Rramewraic fc3r Learning (1990). They would like to beteve that 
research could gukie ctassiocmt psi^^ice, tmt in Uie at^(»K^ of such knowledge 
they mu^ try to make s^ise of things cm their own. This p^r will attempt to 
groiq) the tcacters* commits undor three he^ngs* 

Need fttr more direct links between teachers and researchers 

How -^o teachers fmd out about research? As Qm^in(19S9) pointed ouU 
"Iteachd^I have Uttlemcrtivation fwreading the researdi,l^re^arch kiwwl- 
ed^e sold are unable to lanferstand the di^ m feel that rsseandh is not rde van t 
to tte p^tice erf teaching"* (p, 95). Most of tte {^titkniers intorvfewed 
mentioned thattl^ had little csrnodirectcontSKrt with the litaatur&excq>t when 
actimtly engaged in gradi^tewmk. Motivatimtoreadbf&iciesearchinprimaiy 
sources is much reduced chicc a teiK:h^ is (Hitskfe a umvosity envirroment. 
Even when research journals arc available tteough local teach^ litearies 
resource centrcs, these joimials arc not b(mowed often as perkxlicais 
containing practical ''how to** articles, ^:c(Kding to db^^^ Teachers 
say that tl^ most (rften hear about re^arch second-hwd from a workshop 
presenteTt or even third-hand fnnn scmeone else such as a supervisor who has 
aUendedaconfereocc* Reviews ofresearch arc less likely to be re^ than articles 
in educational magazines wherc research is cited or alliKied to in su|^rt of the 
writer's personal (pinion. Teachers arc [Hesmted wUh research findings, 
prepackaged and prefaced: **Researchsays...** withrote^portunitytochecklhe 
statements for accuracy. 

A basic need of practitioners is to have morc frequent links with people who 
could help them sort out the salient points from all the infcnrntation they arc 
bombarded with at times. Ttey need a kind of consumer report service to 
provide them with ob^tive <?valuaticms of current educational issues, clarify 
the relevance of certain resea* ch findings to their own situations by commenting 
on the quality of the rcsearch methodology, and point out the limitations of the 
studies. 

The most direct links occur when teachers have an opportunity to work 
directly with researchers, though Uiese q>portunities arc all too few. However, 
some teachers would like to be in vcrf vcrf in replfcation of studies done elsewlK?rc 
to see how the results hold up in a tcx^ setting. 



Ahhoi^ te^d^ view edm:itfks^ leseandi as scmicAbiiig reroc^ finrai 
imfep^ic^ aidcmanioiis, Bad a^)diti(»l «:ttivh^ By cmttmt, (m smm 

The mostpcmtive attUuctes towimi exiOTiaUy-developedachiev^ii^ tests 
m kmd amm^ tte grow^ cacfa^ ctf teadiers in B.C. who faave had an 
oppcHttmUy to be invirfv^ ki te^-<te>^k^»n^ ^vities tfiemsdh^ tNvKigh 
saving cm om or cAh^ erf* tte varimis i»ovincid ch^ d^rict ocmimiUees in 
ccMU^tkm with tl^ B.C. Learning A^ssmm, Progjrma siiK^ its ii»;:qMiiHi in 
tte mki-seventies (Mussto & Groer, 1980). 

A recent example occurred during the 1989-90 schod yem the SUKtent 
A^^sesantaU Brm^ facilitided a {mx^ for theexdiai^ of tocallynkvelqKd 
assessnimm^^uds. IncHi^ca^fourdtstnctshsdabe^ysaupacoqierative 
iteiQ-{tevek)im^t pmj^ oi iteir own wd were seddng Mini^ assi^ance to 
c(Hitioueti8»rw(»dk« Agroupcrf^Mathem^tcsoocHdinatorsmHttliei^ 
teaches^ had started to ctevelop a pool oi math items «Hne of whkih had be^ 
used in schools^ unit teles' year-end te^* (^herit^s had bera written by 
gnHJi® of tc^Aers meeting fiPMi time to time for afler-sdiool in-Krvice sessions 
amIoccasicmaUy with retease time iKuvided by the districts, ThecoosdlnaKMs 
h^ SOTie exportoK!© with itera-writii« thnnigh tteir own invdvement on 
various Ministry math promts »kA as im)viiK?ial exams, classrocm achieve- 
ment tests, and provincial nmth as^ssmmts (Taytof & Rc4>itailk2, 1987). 

Thisisffli exampleof ajHojectinitiatcdby the users themselves whkh in turn 
contributed toward a Mmistiy-iniliative, the (tevek^Mncnt a provincial item- 
\mk (Carf)ol.l987). Tte project gathered mon^jntum due to the enthusiasm 
g€9ierated by the math coc^natumand the teachers. 

Need to f^n mse$sment to match new ptx^ams 

Tl^ new direction in B.C. is toward ways to measure the sludcnt*s individual 
level of attainment ai specifK^ teaming d>jecti ves. The math project described 
above grew out of the traditional experteiK^e of measuring individual achieve- 
ment against grmip perfcnrmance. Given tte nature erf die subject mattw in 
mathematics mi the explfcitness of iIk? curriculum guide, it is likely thai the test 
items can still be used by teachers to describe individual performance in terms 
of coiK^epts and skills mastered. For other subject areas, and indeed in 
anifcij^fcd interdisciptiraiy ^roaches to learning, the development of assess- 
ment techniques will he more challenging. 

How will tenors he able to evaluate their students in the new context fw 
learning? How will they be able to meet the requirements of the new appro^hes 
to cuniculum while at the same lime dealing effectively with pressures from 
pai^ttis to rqxfft student irogress? When parents ask: How well is my chiki 
doing? they usually mean: How well is my child doing in relatk)n to all the other 
stuctents? Even with the best informatton cami^gns that the Ministry, districts 



airf schodscOT dc^, it wiB bca kmg time before parsptsfidly pndtoand and 

pie$si0t& cm imfivubml leadh^ 10 

In te»jiei9 ttem^Im wtU to re^ 

tte&ffi^tmig/Iefmfaigsitiatim. Inmk:idly,iniecm3^S8,iiimydfi^^ 
sp^ a good deal of their profes^Hia! deyeto{Hne«t mSf ^vefopmrat 
ln«|gets Uitnxkiciiig tei»:h^ 

nkifiesthitta^uii^atem^ter-cesit^midt^ Mostctf 
tim ex isting teacha-m^ ami {em:f^-sel^3ed left matm^ aie based oo the 
as«iinittic»i that a wfK^ cla^ (or at kast a gnn^ sbKlmfis) has teaiwd a 
particular unit of work. 

In tt^ new situatkm teacb^ will need to be aewiti^ to tli^ at^es in the 
leaniiagpnx:e^^beabtetoiec«^nhseindicrtt»^o^ The i»di along 

which m individual ^od^t's learning irogie^es may be cmtinuous aid 
seqi^Ual, or at times din^uisivQ m wm t^tes^w. In titni, this calls for 
cmricula th^ aie ^t fcmh wUh sufftciem q)ecincity to allow t^cheis to tell 
wheth^ the student is at leai^ leading in the ri^tdtiectkm. No longer can the 
teadK:^ be assumed to be in ix>s$e$$tcm of the "^ght"* aro^^ 
directed classroom. 

With the inUxxlacticHi of the new Primary and Intmnediate {mogiams in 
British Columbia with their iniefdiscipUnary £qpfKm:h to learning, it is likely 
tfmt the language arts: leading, writing, liste^g and spiking, will receive 
empha^ in all areas of the cunkulum* With this in mind, some teachers are 
loddng towan! evaluation in the language arts fc^ clues to hdp with new 
directions in classroom as^ssment. 

So long as measurement practice l^nl mainly toward multiple choice and 
other readily scorabte item fimnats, language ads and English teacters were 
skeptical about the usefulness of such tests. But many t^b^ have been 
introduced to m^w apfnoaches to teaching and evaluating stinient writing 
through curriculum impteuK^nted during the I98(^, ami leinfcm^ed through the 
v<ay successful Young Writers' Program. During the r^utar provincial 
assessments of written exprei^icm teach^ have been inv<rived in developing 
and scoring student writing using both holistic and analytic maiking scales. 
With adequ^ tmining and monitCHing, teamsof tethers at tii^ district levelcan 
also i^xluce reliable results using such scales (Mclntyre, 1987)« 

Crucial to this process was tl^ devetopment df valid crit^, specific to each 
writing topic. Although markers were often in agreement when asked to rank 
order papers, tl^y differed about the reascHis for assigning scores. The {Hxxress 
of develoiMng criteria was, in f^t, a valuable in-s^ice experience during 
which the teachers gained new insights about teaching and even modified their 
long-held personal beliefs* 

The introdiction of the writing i»x)ce$s as a method leaching writing 
meant that the teacher's role in the classroom changed from ihe-CMie-who- 
marks-the-papers to someone who advised, coached, modeled, encouraged, 
ccmmented^ and gave editcnial advice. Teachers who had served an district or 
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p^Yiyire^iat mnriring teama sad thflt tte expcsrience gave thcan grealfr a^BiTOce " 

w^^yrtfmn»dtod»kc{asaPO(m». WiA^i^nrHiiarysitdl^ni^^ 

imj^mns ^le wiU be ininy ^iatkH» ti^i teadKT (»nmH fi^ 

the'VigM.'* answer initwiUlmve to demfflistiatefiainffiss^ With 

Bg»K3Bt&)o<tfpaieiitst]9«^i9C»iaccc»ittd^ 

see a iQttim to a pemus^ve iwf^ aa^b as tte laie sixtks wf^ tenc^ 

dioose ffli kHosynofttic ^po^ 10 evali^kHi. Instep tead^rs will need 

qqxmunit^ to wifffc {(getter (m a £uriy r^piiar baas to de^ 

{rfcrii^ fai ondler to validale dieir own judgonents c^sliK^ woric. 

I^iifllria a^mi^aliKS pointed (Hit that pdicy md^^ 
thattl^co^ofisteasiagte^:ti»rstodeveb9evak^ionmalarialste It 
i^KHild be imlttslood dmt »icA wajc finr beyoi^ 
ua"— whateverfonndiatiatxlQCtnmyfidGe. WteoagrotipofknowtodgeaMe, 
experienced, articulate teasAers gel together on a project they »gu6, discuss, 
criticize,expkjre,creale. What they are doing is describing very finely what they 
are ieffi:hing (curriculum c(mtent); how they teach (instructitMialpactttcs); and 
how th^ know what the stud^ has teamed (evaluation). During such 
<f i<yi'fpri<^« f h«y can focus «i the formaiive purposes (rfeTOtoation because they 
aieiiotreqiiired to diiidi^XHit grading. B<Mhl^;timaiKt»}ministrat(»8 found 
boiefitfincmiwcxidngtc^^h^ cm evaluatira projects. It makes ec(»H)mic%nse 
to encourage these c^ppcHtunities which eiabte teachers to be more aware of the 
connectirais between curriculum, instructi(m and evaloaUm in the classnxsn. 

Summary 

The teachers and «iministrators intcrvfewed all agreed that increased 
c(mtact between re^archers and {sactitioom would be beneficial, particulm-ly 
in light of the new curriculum changes in B.C. They see three ways in which 
educmkmal measurmnent cwkl assist them: (1) providing in-sCTvice m 
evaluatimi to m^ teachers mote knowledgesbte in sdecting iq){Mx^riate 
evaluf'^Nimaterialsandtn interpreting testresults; (2) assisting teacher groups 
in dtevek^ing new evaluation materials to match tte new curricula: and (3) 
involving teachers in research projects in local settmgs. In these ways teachers 
could become more famiUar with the language used in research studies, more 
able to judge the retevance of results to their own siti^(»is. and mcHe conftdent 
in the practice of teaching and evaluating their own students. 
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Emerging Needs of The Practitkmer in B.(X Classrooms 



AtoiR. Taylor 
Coquitlam &diooI District 

Background 

Chmige Im becMO^ tbe ncmn in today's so^y^ to U seems that people 
scanty Imve tune to i^jutt to kmyf^n befcne tim imt mives cm the 
sc^ie. Nowh€ieistlus{d^KfixiefKHimo[eevid^thiia 
exploskm cunmiy imdefway. It is evidoit, fm exainpte, thm tte am<mnt id 
tfichnkal and sciemifk: data is growing at Atpiesemit 
(totd^inte^thanffveymsanditisinedictedby 1996 it will dmd>leeveiy 
twenty months (McKerlfch, 1987). 

In (baling with this deluge ctf informatim^ the imp(»taiK:e in the ctttrkulum 
of d;etfa processing ^i\s (the ability to funl infmnatkHi, letrieve it, classify it, 
intef]H)e^itandiep(Htit)hasbe€onK?drmi^^ CHterchwges^with 
impluations fos the deveIoi»nait n^w cumciila, have resulted in the fdlow- 
ing tuftsin focus: fincMn tewb^ to ternncr* cmtent to im)c^ ami 
active teaming. Each im tiemendoi^ implicMions fc^* the educational system 
in tenns of its (»tgantz^n md the ^veyry of its product. 

Ttm purpose of this paper is to di^uss »nu5 of the emergent needs of the 
practitionear which can be met by research in dealing with these changes. In 
^Idressing this issi^, it first e^listes the franwwcsfc within which llwy have 
evolved. Given that ccmtcxt the papar iMOCceds to identify those aspects of 
change which relate to cuiriculimi andsuulentevaluation. $€x:ond,itfocuseson 
a number of specific questi(»is which i^ed to be addressed in the area of stuctent 
evaluation. An aiproach, in which ll^ tether plays a meaningful role in 
research, h suggested as a means to address ll^^ questions at the classroom 
level. 

Setting the scene for chimgc 

Changes in technology, Uie environmOTt, and family structures and priori- 
tfcs, in a rapidly changirig world are among the factors which gave impetus to 
the government and the ediK^ational community in Briti^ Columbia to reflect 
on thepurpo^ and outccmies of schooling. In response, the Province of British 
Columbia in 1987 initiated a Royal Commission on Education. Recommenda- 
tfonsfrom that Commission dealt withawiderangeofissues,anumberof which 
related directly to the curriculum and student evaluatiw (Sullivan, 1988). 
Among those with implications for researchers and with dirwt application at the 
classroom level, were ite following: 

... (tevdopmental criteria, rathcar than chronological age, be used in selecting 
the educational placement of children entering school (p.28). 
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... eaal^ schools and sdKxd dimim lo meiiMt tmgiaded prknaiy 

... provide feamois ^ ^i^aa^esstomiiltigr^^axisj^ 

Furtl^ to tl^ leccHnmoidatiof^ 
of EdBcatifm sitouhl provide giikiaQce on standards aod criteria for 
t^^lim to i^irfoy e^^dmti^ ^d^ts^ perfoitHaBce^(p«l 1 1). 

In T&pmx 10 Use CcHiimis»n*s findiii^ the Nfioi^ Edi^aticm 
dev^^>ed tte Yem'2000Pl^4)e3'(Mini^crf'Edi^at^ 
fOTfoUow-up^Uviti^ Itpn^x)sedaciifrictthimafid8s«ismn^f^^ 
ihd British Columbia schnA system l^&sed cm a mandate ami adescriinion c^ti^ 
'^educated citia^"*. Hie document, which is ^1 m diafl fcmn at the time (d 
writing, attemiHs to trai^Iaie many of the intrats of the Gommi^(Hi's findings 
into an (^)^Btk)nal OMitext* 

Many of the changes proposed in tte papei* require furth^ (fevelqHnent at 
the hamb of the classroom pr^titfc»ier mi tlw r^earcter before effective 
implementation is possiUe* A ti^ of gnngx^ uncter two categcmes: 
curriculum features, ami evaluation and reporting, are listed below. 

L Curriculum Features 

It is proposed that the curriculum : 

- be learwr f ocussed- to be "develqrm entally aRKopropriate and sequential, 
allows fw continues pnogress, provides for s^lf direction, and is individualized 
as miK:h as possible** (p. 10), 

- be organized according lo four strands -humanities* sciences, fine arts and 
practical arts 

- include a common component which incorporates elements of all four 
strands 

- emphasize intended learning outcomes rather than learning activities 
2. Evaluation and Reporting 

It is proposed that die evaluation and reporting of student progress include 
the foltowing features: 

- be critericm refe^nced 

- include a variety of asi^ssmenl methods 

- include continuous progress 

- focus on learner profiles 

Similar directions alj^ lived from studies undertaken in ^verat other 
jurisdictions. The State of i^unnesota* for example, established an Offtc^ of 
Educaticmal Lead^ip in 1989 to estaUish an agenda f(^ youth into the 21$t 
century (Ofiice of Educational Leadei^ip, 1990). Many ofthe elements in the 



itoia«tevekmedwaBsiiiiitoloaio»taBri^ Atlhescho^ 
d^tev^comiHebeBsiveplaiiswiihstaito 

Ontaik) and Ihe Ctoqmaam Scbool Board (1990) to 

Qtesdons to he Addressed 

AsteachOTauempttotaiplen^texlCTfflvechaigefk^ 
2000I^,a&lilk>Mlinfaniaaooisesse^ Although 
some are not the extent of proi»sed cAange and the artk^^ 
tioi«.caiisethemtobcmcHepi«ssiiignowttianever. Tlieseisuesandquest^ 
m ffoapei umfcr two cate^Kiest curriculum and instniction, and sludenl 
eva^ttion ami rqxming. 

1, Cuniculum and InstnicUon _ 
Uisessenlialthatchwjgwimjposedbyllw Ministry of Educauon in Uja^ 

of cdiriculum and instiuction be based on solid research findings. AKhouiih 
woik has been done in some areas in which change is contemplated, there 
remains miKA more to do in onte- to gain a clear iiraferetanding an^ 
Many of these issues were listed by Co^ (1989)> his Foreword to l^^ 
ASCDYeaibook on cunentcogniUve research. He identified thefollowmg 

questtcms as most imessii^: 

Which of tools of inquiry are impcwtant and why? 

Why arc modes of inquiry and thinking important in understanding and m 
teaching schocd 5ub.^cts? . . ^ . u-^^t 

How do modes of thinking intersect with the knowledge base in a subject/ 

What in^tional processes best develop subject matter concepts in stu- 
dents? , . . . 

How much time should be spent teaching various concepts and at what 

developmental ages are they best taught? . , ■ f ^ 

How arc these OMiccpts and modes of inquiry organized to be remlorced 
throudwHil the curriculum and the school? 

Howcan wehelpprcserviceandins^ice teachers understand the concepts, 
modes of inquiry, and thought processes, and how to teach them? 

Which of the modes of thinking and tools of inquiry are genenzablc to other 
subjects and to daily Ufe? . 

How can we measure and report growth in thinking abilihties? (p.vu) 

2. Student Evaluation and Reporting 

As a result of the proposed shUl toward the student as a learner, many 
questiOTS also remain to be answered in the area of student evaluation and 
reporting . For example, implemenlation of continuous pivfress, the use of 
student profiles, and i^xming on what students can do, aU have significant 
implications for procedures and practices in this area. In an attempt to come to 
grips with some of these, a discussion of issues related to criterion rcf«c^i^«; 
measures, methods 
techniques follows. 



grips w*u> <>v-«- - — » . . c 

measures, methods of authentic measurement, and varieties of measurement 
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<i). QitedonRef»em^MM»Hes 

Movemmt away finmi mrnn referenced mrasses kkI UmmA expe^^ 
^^Klards of pedormance is c^Misid^ wiA &e oc^Hiqpt of sAiutent focu^ 
teaming, Iid^r»tinUiis^ft,howem,i8 i^fcf more^eoivetecimiiiira 

whicb 10 set standffids* it is essen^ 
tntbeirrepcEtii^ctfsliKi^proffn^ Fcnrex^ple,cteiaiy cabled clescri^ 
wbfch de»:ribe ^iKtett imsgn^ 8k»v a c^Ri^ai^ 
articulated; ^edive m^kxis imffmg tead^r ^tkul^icHi to be 
estri>liMied;aiidimx2^ii»teswlu 

(kimnifiedL The l yp t m H la te iro (rf iMymative infcHTimttoi also needs lo be 
incoqxHBfied into tfiis pnDoess, od^rwise tent^ jmlgm^ts bs^ in ctoig^ of 
straying apart fnmithc^ctftfieircoUeagws, C^K^ted^i^^iiiArim^ 
aie (teve{(^}ed» th^ lemains a cswkt need for d^ training tei^dNeis and 
administrators in a$s€^mcflt [HBCtfees. (StigginSi, 19%) 

(it). Methocb of Authentk; Measiffcnmit 

In a recent artkle, NfcLoui (19$K9 defined authentic measurement as 
'^rfonnance m mtanbigful tasks in a rec^^lsaMe cmtexf ^(pJ8). He 
provided as an ewmple, use of a ^itftent ?c»rtfdio which can be used to pmvide 
a cumubttive lecwd of stwfent idbibvement fc» a wide vari^y of aciiviti^. 
Odier e;(;amples c<»ild incli^ the use of interviews* stq^by-step jrocesses 
employed by stiKtents« and "on the pV* s^sessmodts tor suideitfs engaged in 
activities out of the classroom. There is need io this area fc»r tte develqnnent 
of a number ways to measure stu(knt activities in difl(»^t settings mi fac 
a variety of purposes. As these techniques are <fe\^k^)ed, several ancilliaiy 
products are needed, among them sQ^propriate in^mcnts for the collection of 
infonnalion,procedums for interpretation of that data, and methods of tcporting 
it 

(iii). Variety of Measures 

For years teachers have been encouraged to use a variety of measures when 
they collect information about students. Most of diis information, however, has 
been related only to intellectual develcpment. In the proposed changes teachers 
will al^ be expected to repcvt to parents in meaningful ways cm the Ingress of 
theirchildrrninbumanandsodaldevdofmiratandittcareei'deve^^ This 
expectatfo?! gives added emphasis to the need fcr bett^ methods of evaluating 
attributes oftheaRectlve domain. In additi(Mi,nK»ewOTk is needed on process 
evaluation and critical thinking* Some of this work may be done through the 
refinement of oi>servationai techniques and the use of holistic measures. 

From the practitioner's perspective, direction rdated to the preceding issues 
isessenlial . Forexample,inthenewprogramitisexpectedthatinquirymethods 
are taught and evaluated; teaching for thii^ng occurs; a variety of instructional 
tcchniqi^s U> match needs of the bamer are employed; and an interdisciplinaiy 
approm:h is used in teaching of fhe curriculum. 



Tt^ex^ofimqiosedcinkuhimc}^ Not only (toes it 

invcdve change in cc^tent mi hig^ tevels ctf d^nitim, hut it also has 
imi^iicatkms for classroom organization, groui^ng ctf students and the use of 
diffi^i^ teKhing strategfes. 

To effectively imi^ment change of this magnitude the classroom inm^ 
tkma'ismde^)^:tfeiie^(tf^icotirag«n»tam!diiecti^ Yetr^aich seldom 
n^cis tb^ neecb, Fw^ianiple.POTy-^hd(ka& Allain(19^ 
^nce ti^ immedi^ ctf day-tOHlay lii^ 
they tend to nudce dedstcms based cm tl^ own expertences ai^ c^ 
rather Chan c»i thc^ctfexix^OT the flndiii^of le^^ 
it is ncH the ose that te^^h^ do iKH OT witt not ^^y Hndinp 
ttieir practices* but that they cannot This peicqption is si^^XKted in findings 
fn>m Canr and Kemmis (1986) who sugge^ that teacters tend to view research 
as an ^c^eric m^tivity having little to do with ev^day matters. 

At issue is the eflectivencss of the Kq>-down model currently employed in 
much of re^arch* This model may be ineffective due'to the illation of teachers 
in the classax»n, who ofim woik without cont^t with colleagues or frequent 
supervisiOT. Tye and Tyc (1984), for example, conieml thau 

.... We continue to mmidate changest evc»i though available evidence 
ov^^helmingly indicates thai refcnms imposed from the top are incflecU ve in 
bringing about desired changes in schools. 

The need for tte direct and meaningful invDlvem<^t of tea:hers in rcsearch 
was also called f(^ in recent work dme by McDonald ( I ^9), Gage and Berliner 
(1989), and Atkin (1989). In a recent article, Atkin (1989)claimed that teacher- 
conducted research must be seen as an important respcHisibility of the teaching 
profession. In taking this position, he states thai 

The progress of meaningful reform will be stalled until teachers emerge 
from their marginal positions in the research community and become full 
partners in the conception am^e conduct of educational inquiry.(p.205) 

What direction can research gain from these otecrvations? To begin, it is 
suggested that to have an effect^ researchers should be as concerned with the 
implementation of their findings as they arc with the study itself. CrcKS (1987) 
believes that the active involvement of teachers in research related to their own 
praticcs is essential to bring about improvement in learning. This position was 
supported by Tikunoff and Mergendoller (1983) who saw the involvement of 
teachers in conducting reseach as imperative. Based on this information it is 
recommended that ijrcfilcr attention be given to the collaborative planning and 
implementation of research between the researcher and the practitioner. 

Summary 

As reported earlier, the purpose of schooling is .shifting in an attempt to meet 
new sets of needs and expectations. Basic skills, for example, have expanded 
from those only related to numeracy and basic communication to incl ude critical 
thinking, decision making, flexibility, and understanding and tolerance of 



designed tteiri»tx)m:t« ll^ctHTteuhiml^beccmienKmi^^ 
(^^tspednc;it focf]^m<»iecmthmld^ 

setective in team erf tt^.^^ matter ccHHaii^ in it; and it emphasizes 

Stml^t evahiatkm is an integral omipcment dfthe ediKSlicn^ i»€N:ess ami 
must be fair, ccmsisientaiKlmqqxMtiw. AssK^hitmn^ber^pnisivetofetdted 
ctenges m the cucricuium* It is impcdam also that student self-^teera be 
pn^ted UmMigh ti^ indiisicHi of tasl^ thM ^iKl^fs cm be OKX^fut 
Gcmnan (1989) emphasized UieimpomiK^e of this fe^tro in saying, 

Evaluadcm is a two-edged swofd that can cailmice and be ccm^riK^tive to 
student li^ymingandp^r^ality development * cnr it be destnictive to stucfent 
I^tming. The ctoice is ours. (p. 15) 

Shifts in the curriculum, however, have left a void through the stosQncc of 
aipnopriate teaching strategies, and evaluation technk}uesai^in$(mments with 
which to measure student progn^ fcH* many of the intern^ CHitccHnes. This 
paper ii^lified a numba of n^^ in these areas feced by the ct^i^^ 
in attempting to implement change. Hie role of research in this matnx is the 
development and validation of appro{mate pnoceduies, techniqm^ and instni- 
ments« AraOTg them, more effective means by which to measure higter order 
thinking skills and attributes of the affective domain; devekpnent of appopri- 
ate criterion methods to assist teachers in determining and articukti ng common 
expectations; and more effective techniques in process oriented evaluation. 
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Grounded Authentic Assessment and Teacher Education 



Ttomas O* Magtike 
University of Alberta 

Introd$4ctim 

During the past two ^cades thcs^ has been a sub^tial imrrease in the 
anKHmtofext^naUyimpo^as^ssmemof^tui^teain^ Mostfrnmncial 
(kpaitments edi^atim have intuited testing piogranis u^g various com- 
binati(Mi$ formative ami summmive tests to nKmittH* ffi^hkvemrat, to proviite 
indicators of cdiK:ationaI quality and to int>vtde emi (rf school ceitification. 
Recently, the Council of Ministers of Eduction has called fc^ tem^ to create 
a cross- country as^ssment of litenK^y. BeycHid tl^ inovincial and naticmal 
exams, some regions have participated in the p^^Kiic admini^ralion of inter- 
national asses^ents. As we move into the 19%*$ a rc^wgeiK^ fiqp-down 
testing with its conccHnmttent ccHupariscms of stm^ts, te£K:ba:s, jurisdictions, 
piovinces and countries is refocussing our educatkmai dTorts. In this paper I 
will lemind you of som^ of the negative CMsequences of this, and suggest some 
strategies (ot r^ewal. 

Among the several research sliands in tte cOTtempoiary literature on 
achievement measurement that relate to both the consequences of provincial 
assessment policies^ and to the suggestions for change, two were featured in last 
year's Victoria conference m classroom testing. The "grass roots*" researcher 
represented by Wilson's (1990) pq)er, sedcs to understand assessment through 
investigations of current tether practice. In the next section of the paper, I will 
use these investigations to show how provincial activities influence instruction. 
The "improvement of pmctice*' research, focusses on the validity i^t^ of 
achievement assessment One form is discussed in McLean's (1990) pa|»r as, 
^'authentic assessment."". I shall use this line ctf research in the second section 
of tile paper to explain what assessment ought to be like. In the final section of 
the paper, Til talk about implications for teacher education. 

The First Part - Influencen of Assessment Practices on Gassrooms 

In his study of teachers in Ontario and British Columbia, Wilson (1990) 
found that his modal teacher used completion, short answer or essay questions 
rather than multiple choice questions, they borrowed or edited from other 
sources if they were in elementary schools, or built their own instruments if they 
were in secondary schools, and they carried out assessments for three main 
interrelated reasons: to determine marks, to check stu(tent progress, and to have 
students practice what they had learned. Near the end ofthc paper Wilson links 
teacher practise to government activity: "What actually happens in our sample 
of teachers is a tendency to imitate the provincial government in virtually every 
instrument*' (p. 16) And later, "The evaluation of student achievement by 
teachers in classrooms occurs within a policy and procedural frame woric that is 
largely determined by outsiders"* (pl6). Wilson"s findings support the widely 
held belief that provincial testing programs have a profound influence on 



A quick map^ fnun the ATA Mag^^ti^ (a |nd>l tciOtfm the Albcm 
Ts^t^* Ai^ids^) wilt givea fiavrarcdr (he Afteitaconip;^!. In the Maich/ 
April is^ie» N^neifa ffiid Sffinirodra (1^) gatven^ 

with t^dfeos, cla^ccHD ote»\^icms« fimn the c^wikNis exf^essed by 
te^^t^^aca(temk»famil]arwiththeAlf>^s^^ Inthdrcoiidtmm»the 
authons ^i^lSy no m^uis si^mld the re^ifts erf »K:h toting he puSAk ized (m* 
used ^iitdtcatci^ of school OTte^;^ (Imciie^gly ^KMigh 

tte i:^pc^ was written pri(»* to tt^ Edimm^ 
public test lesiilts fw e^ sdKX^) N^i^tb aiKi S 
povimaal as^ssi^ts ^ in i^ir^^ilar tte gnKte 12 diplcMna examinations 
repiesent a politkal ^qKoach to imimving tte quality of educaticm, and wfmt 
isieally needed isOT ^iK^ational q^»t^h« Th^ ^knowledge (implicitly) that 
th^ as^ssment expertise of many clai^mKHn teachers is not high tnit that it could 
be improved, and in this improvement lies much of tte solution to the validity/ 
authenticity problem. 



1 . Diptoma exams contradict ihe belief in continuous formative 

2. For a centralized testing program lo work* tests must be nomied. 
Ncnming a^umes similar cisssroom experiences that span, in this 
case the jffovince of Alberta. 

3. Because there is pressure for teachers to teach them. Diploma 
Exams guide the curriculum and detennine what is important at the 
expense of other goals. 

4. Diptoma Exams substitute content for context 

5* Test blueprints that are prepared for Diploma Exams have content 
area and cognitive level as their main dimensions. This classification 
is det^mined by the examiner, Init the kind of thinking required is 
actually determined by the student 

6. Diploma Exams test for a very limited portion of ih^ cmriculum. 
They are geared to measure those pieces of knowledge most easily 
measurul 

7. Diploma Exams do not correlate from year to year because the 
questions are revised* some new new questions arc added, and the 
students from year to year differ. 

8. Comparisons between individuals and jurisdictions arc impossible. 
Two students who receive the same score do not necessarily 
understand the materials equally. Oftat they do not answer all of the 
same questions. 

9. Test results cmly correla " .vith future results on similar tests; they 
are not accurate predictors of future success. 



Figure 1. Problems associated with diploma exams (From Ncmcth and 
Samirodcn, 1990) 
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Rk* tfie^ r^scMis r^>kmia Exam tesidts are ckA gooi iiHik»tt»s of the 
quality of edm:^kuL 

In British QAuidHa ibe recoid^ 
examinatkms m educatkm (AmieiscHi, Muir, Bateson, BladaiMHeaiidRQgefs, 
1990) cdiected opiiii(»» fitnn tea^ 
Although the analysis im ciumy^tbe tr^^ 

impc^ exams are vtewed as ccMistraining t^K^ing and temiing (HBCtlce, in 
tenns of what is taught, how it is taught and how it is assesi^. In shcHt, the B.C. 
di^ aie generally consistent with die le^ fonnal Alberta probe described 
above. 

What are we to conclude bom this informaticm? 

L Provincial asses^ent {Hograms and in particular summative 
have an impc^tant influence on instnicticsial activities iiK^luding clas^xxm 
assessment practice and th^ directi<»is are tka always benefldal to students, 

2. The assessment ^Is of many teact^rs are restricted at least insofar as 
they demon^te use of a ^wiety of technique. 

3. De^ite d>$ervation 2, there is an intuitive base possessed by tethers 
upcBi which better assessm^t prxtic^ could be built . 

4. While educational assessments at all levels seem to be purposeful, they 
aie not obviously driven by any theory beyrad tte important a{^)eal t(^ social 
concern (for example, accountability). 

The Middle Part - Qrounded Authentic Assessment 

In this section of the paper I would like to briefly reexamine and reinforce 
the school of thought that makes validity the fundamental concern of all 
achievement assessment activity. Much has been written about how the 
construct validity of achievement tests can be improved. A lot of this material 
is written from a cognitive psychology perspective (see Snow and Lohman*s 
excellent review in Linn, 1989), but there are also important directives in the 
cjmcutum literature. Haertel, (1985), refers to these as competence theories, 
and although he classifies them as psychological I would call them cturicular 
because Uiey /\ ..always involvea detailed analysis of specific curricular content 
(p.28);* 

In his paper, McLean (1990) reminds us that to be "authentic/* assessment 
instruments must reflect, "the knowledge and the jsocesses that form the basis 
of the subject mailer/' and the, "conditions under which die ^hievement 
normally lakes place,** (p80). Although not mentioned by Mclean, the entry of 
cognitive psychologists into the achievement assessment field (for example, 
Embretson, 1 985) can be seen as an important step toward authentic assessment. 
It is a breath of theoretical freshness in a field dominated by the mathematical 
mustiness of item response theory. 

Because authentic measurement must reflect the context of instruction, 
McLean warns us thatit is "...impossible to produce measuring instruments that 
can be used in the same way across a counuy , or even a provi nce(p.8 1 )/* This 
warning flies in the face of the political forces that seek to implement and extend 
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<^mip»:alii« fe^faig {m^rains M the iat^^ 

In a fi0C^ Naii(7 Cote ( 1 990), pointe out 
is»^tb^I^ld^fettevalidas^sIIlrat(tfffi:hi^^ She refers to Bmidy's 
(19SS), The Uses cfSckai^ing as an exampte of tH>w i^iitasq^^ of ^iucatkm 
have inqxmmtt m^s^es fin* pei^te in cmr fkld. FkY»qMed by Cde» I lead 
BztHidy's nrKmogrqrfi and became coovim^ed that mith^uk: assesmioit is ml 
siiffic^t if we are gohig to lestrkt the te^ "^fiuthful iefkction''of the 
km)w^l^ ami processes tl^ s^ject matt^. It seems to n^ th« valid 

in tte i^bok)gy of ctevek){mient, temning ami iiBtfiKrliai. Th^ vtow is 
consistrat with Mess^k's (1989) extenskm of constnict validity to include the 
consequ^Uial ba^ for int^i^etaticH). 

In oudining his aigunient for general education, Broudy (1988) lists four 
uses of schoolii^: lei^icative, ^Ik^atlve, as»ciative and interpretive. The 
rq>lk:ati ve use of knowiedge refers to the use f^ts and principles just as they 
have been learned. In our parlarce we would tie this to Bloom's (1957) 
knowtet^e (*jectives. Broudy points out that there is miwh that we Icam in 
school that is soot forgcMen in the fwm in which it was learned. Replicative use 
of learning dQ)eiMls toson^extenicMilN? individual. F(»-exampte,Inolonger 
rememb^ much (rf the Russian vocabulary that I learned in oider to satisfied the 
requiiements for the Ri.D. On the other hand I have used the multiplication table 
so much th^ it has b^cnne ovaleamed. 

The applk?ative use of schocrfing refers io the direct appIicaticMi of facts w 
I^inciples to new sitimtioM. Much of technical training and apprenticeships, 
and some of im>fei»icMiaI educatbn are directed toward the ^licative uses of 
schoolmg. Attempts are made to show suidcjits tow to structure, categorize or 
recognize situations wtere facts and priiKiprfes can be applied as learned. 

The otiKjr two uses, associative and interpretive, refer to how we use the 
associations and cwnot^ns that are built up around concepts and principles, 
and bow we make translations and abstractions in octter to live our lives, Broudy 
points out that the real value of education lies in the latter two uses of schooling 
and much less on the former. However he justifies the former by showing how 
Itey contribute to what he calls the "allusionary base,"* the conceptual store- 
house ofimplicit knowledge that each of us possesses. As he notes (p- 21),*The 
associative resources provide by schooling and experience plus the interpre- 
tive repertoire of ccmcepts and images consititutc tiKi allusionary base. The 
resource of the allusiemary base are not used by simple replication of this or that 
school learning, and their adcqu^y cannot be judged by tests of replication or 
application. On the contrary, the success of general education is to be measured 
by the depth and quality of the allusionary base." 

The point of describing Broudy 's ideas is that if you accept them as a basis 
for general education, there are dirwt implications for the practices of assess- 
ment, both at the classroom level and externally. The philosophy should drive 
the instruction and assessment, and conversely the choice of assessment 
instruments should give clues as to the philosophy in operation. The complaint 



of te^^imaiid theobs^vatkm^ 

i^glectthea^(xd^%«aiKiiia»[im^vei^ They (the b^){miiUL 

a pictiTO of sctKxds th^ »e cfe<^^ 
ecbic^ioiL Oiff imikxsiEduate nrnasiirm^ 

1956 view of eduoitioiiafob^Uv^ 1956) ami at wottt based on 

tf^ sparse logk^tty |x>sitive dtctales of ifi^ bdm^ 

tte naige of as^san^ts. But what is tik baste ctf this extension? Tb^ is a 
philc^^y that is imi^it in this treiKt ami I believe timt it mvtA be npliraied* 
I remind you thm ai^Bemient, even authentic ass»m^ is mrt vali^ t^tral . 
Different irfiii(^o|rfucal starting pointe yk^ld v^ difTeient mitli^^ meastaes. 
We must encourage assessment cmstnictc»is at all levels to make their 
foufKlational beliefs and assumpticms public AuU^ntic assemic^ts that fux^ 
derived from such a netwwk could then be described as being grotmded 

Consistent with the need to grcHmd aitfhentic ^ssessmoit in a frttilo^^iGal 
framework; is the impOTtance of designing ass^smoits that sors^ instruction, 
and using the as^ssment process to i^omote useful and respoi^ible self- 
evatuaticHi* 

In a recent review of the research on the imp^t of classrocm evaluation 
pnK:tices on students, Crooks (1988), reminds us that tte quality of stmtent 
learning is greatly influenced by the nature and fr^uency of our assessments. 
The thmst of his argument is that asse^ment must cmtribule to and nc^ be apart 
from instruction. In our terms, groumled authentk; assesi^nent shcnild emx>ur' 
age de^ learning, it should provide effecti\^ feedback, it should promote 
appropriate standards, it should encourage indepwitfent leaning, and it should 
reflect the important goals of schooling. In ^rt. Crooks tells us to seek 
instructicMial validity in our classroom a^ssmenis* Clearly, there is a coher- 
ence among the voices of Broudy, Crooks, Cole and McLean, 

To summarize the middle sectim of this paper, grounded authentic class- 
room assessment must be surrounded by and derived from an approach to 
education that is based upon a phftc^sq)hy that is suj^ported directly by the 
educators and indirectly by the socio-political views of most of the nation *s 
citizens (social fidelity), it must be authentic, it mtist suf^xjrt instruction, and it 
must encourage individuals to take responsibility fca" their own learning. 

It is clear from part one, that provincial assei^roents have been weighed by 
many in the educational establishment and found wanting. I am not convinced 
that the picture within classes is much better What can teacher education do 
about it? 

The hast Part - Implications for Teacher Education and Teacher 
Educators 

I believe that there is a widespread feeling of powerlessness among teachers 
when it comes to dealing either with measurement issues in general or with 
externally imposed assessments in particular. This is in spite of the fact that I 
have never heard of a Canadian teacher being fired for the test performance of 
ht's or her siudcnls, and in spite of the fact that provincial exams are usually 

ulJ^. 
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cie^ by rqare^uative Mhm. If this fi^mg of powettessim is as 
pravasive as I AM k is. th^ we need to do scm^hing to onpowo- tsstchsxs 
beginning within our teacher wlucation programs. 

In niaqr faM&Mi(H», ^bsmtoi stiKteHs are iK^ requhed to take a coui^ 
ciasaorai ass^asent so k is not suiprisii^ that they know very little ^xnit 
c(mc^ like grade oiiuval^Ke, validity and «nar of measioie^ With a 
t^op^ fcnowtedge base, some wdl-hfmed analytical ^Is, and a little oonfi> 

dm:e, te»:b«i8 could lead tte ddiate on ass^smem and sandards, and not 
tl^n^ves lesrting to tte imp(^tions (tf oUkts. 

We stould b^ by helping our edtK^on students to (fevelop provisional 
opeiatiaml iMosc^es of educatkm. Aida Cole (I^) {novides a good 
illu^ration of how the pernmal phik^t^'es of four beginning teachers 
devetoped during their first year of tca:hing. It seems to me th« their coUecfion 
aiKi use of assesane»t inform^cHi shoukl be based on this porscKial i^ilosophy. 
Teacha^ with well explicated fcHmdatim can respond effectively to external 
infomaticm, they can woik with parents, and they can tteir assessments 
nusemeaningful to students and to instns;tion. Atpresent,whCTe measurement 
cau^ exist, they ^em to be steeped in the technology of educational 
measurement. While this is iroporiam, I do na believe that it is the best starting 
point 

We need to teach students grounded authentic measurement strategies. One 
example of sudi a strategy that is consistent wiUi Broudy's i*iilosophy trf 
general education (although it is not based on it) is Biggs and Collis's (1982). 
taxon(Mny fcM-categaizing stiKtentresponses to varicHis educational tasks called 
the StnKture of Learning Outcomes (SOLO). Untter (his system, learning 
outcomes are assessed by placing them into one of five levels: prestructural, 
unistructucal, muliistructural, relational and extended abstract. 

The categories are defined in teims of the amount of memory capacity 
required in order to make the re^nse, the kind of relating operations that are 
required to produce the response, and the d^ree of consistency and quality of 
closure displayed in the response. Preoperational responses require little 
memory. They may be adcnial of the question or simply repeat the information 
ccHitained in it Often the responses reflect a lack of engagement between the 
stuttenland the task. Unisiructural responses concentrate on one relevant aspect 
of the task cw concentrate exclusively on one part of the solution. Closure is 
premature. Stuf^nts making multistructural responses treat aspects of the task 
independently, and although the memory demands arc higher than at lower 
levels, no attempt is made to integrate the dimensions of the problem. The result 
is a response that may contain inconsisitencies or in which inconsistent infor- 
mation is ignored. Students who provide relational responses attend to various 
aspects of the task in relation to each other. Integrating themes may be used to 
organize the result Inconsistencies arc addressed, but no attempt is made to go 
beyond the boundaries specified by the task. At the extended abstract level the 
task is treated in a context that goes beyond the immediate information. 
Generalizations may be made to other situations of this type, but care is taken 
to describe the domain of general izability. Conclusions may not be definite, but 



amy be qiffllir^ to aOcm ftff k^^ally |X^^ 

If,likeBroifiIy» wettc^^soc^vemdimptic^vetisesafs^^ 
imiXHtaiit* we would like our students to c^xsate A Ite telatk»^ ami 
eiu^ded ^^tim:t levds c£ tim Utxcmmy, l/bmova^ m wasims we would 
ccHistmttly try to move stiK^ts fixm lom^ tevds to htgh^ ci 
op^ation by showing tfa^ how th^ can use thdr dlas^^ 
higher fevels of discissskHi* Finmitli^stiictenitpa^$pective,tl^ 
gives a ^ructuie for self evaluation* Itprovfetesthrai wilhasc^th^theycan 
use as metacognitive tod to raise their level of tlK»^ 
t<^ic. 

The SOLO taxOTOiny isn{Hapan^::e8,n^to'iiisMexam{deofanapfO^ 
to measurmic^t thm ccHild be af^Ued in many ctieumst^K^es fnm povim^ to 
classroom. Provincial assesf < ^^nts (hat rqpcHted r^ts 
would act in a manner that is co* ^stent with classromi instnH:tion» and not be 
as fc^ign (o it Criticisms of external examinations thtt are tmsed (Hi the 
constraining features of these exams would be less supfKmaUe becmise students 
with a wido* knowledge (or docpor allusiof^ base if you pefcx' Broudy) 
would have a greater likelihood of pericmning at a high^ level. So it is in the 
interests of both teacher and ^dent to develop a wide expm^tial ba^. Yet 
knowledge of a common set of speciffe facts would ncH t>e highly r&infoiced. 

An example of the u^ of SOlX> in assessing skills in high school scfem^e is 
provided by CoUis and Davey (1986). I have used it to evalimte variations on 
a pn^TMn for gifted students (Maguire, 1988.1989), Biggs and CoUis (1982) 
provide examples of use in histc^, geography, mathc;matic3« English and 
modem languages. They show how the tax(momy builds on instructional 
theories and how mstniction can be built from it. In $hc»t it ppvidesa reasonable 
example of an approach to authentic a^i^ent that fits within a philosophical 
framework and contributes to the instmctional process. 

Cmclusion 

Therearecompeting demands for achievement information. We must begin 
to develop trust among all levels of the system so lhat ourpracticcs arc consistent 
and fcad toanjropriate student growth and development Obtaining an eflfective 
evaluative thrust begins with making agendas public. If achievement testing is 
to be used to change teaching pr^tice. then tet us say so; if it is to be used as a 
basis for fundingt then this too should be made clear; if we want to encourage 
the equitable distribution of scholarships through our diploma exams, then this 
should be made clear. At tl^ present time, 1 believe that there is much suspicion 
about the purposes of external exams. If we can make purposes public, then we 
can begin to direct developmental efforts towards agreed upon goals. Trying to 
make single assessments serve many masters (some of whom appear after the 
fact) is simply not wcMrking. 

A final word must be said about "accountability.** There seems lo be 
ebelicf lhat external examinationsarean instrument of rccountabilily. Thcierm 
itself isnowsooveruscd that it has lostmuch of its specific meaning. Mweover, 
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ttoe arc claims thai ihe public demands ao«sint^)l% to educaiion, and that 
these d^imnds can bended by extenialexffinb!^^ I believe that tfiis is 
laigelyjwMstated. T1ieiesdtsoftheE(faiioiiKmFb!^SdioolBoaid's(feds^ 
tofmhl^schodiesidtswasmdiffeiiram bgraterated two letteis to the editor, 
niereare few people who me interested in ihe "ffixountability as achievement" 
ph^JomeiKHi but their apparont oaccess has been due to their ability to make it 
seem as thoiighth^represem a grass roots inovemenL We have uncritically 
accqjied studies of American schooling as being applicable here. I see no 
evidenceofagreatpopularconceni. If they arc ikh asked, the members of the 
public wWcOT raise the issue. Ourpartrcqjation In international studiespiovides 
interc^ing IbddCT for the innca-p^ of the newspapm, but once we know that 
we sand above Bttifdna Easo and betow Japan, agood tire fire can knock us off 
thepage. Most of the presawe for external assessments in Canada comes fioro 
within theeducatfonaleslabUshmcnt, We can change this. If we can strengthen 
our teaching population by providing them with better assessment strategics, 
and we devote more of our Ume to research and developmefit of grounded 
authentic assessment instruments, then we will end up with a more facilitating 
system of assessment I think that we have the ability to do it. Do we have the 
will? 
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What Skills Do Teadm Need in Educadonal Testing? 



Ronald K. Hambtetro 
University of Massschusetts at Amtem 

EdtK^itMial testing in the 1990s is going tokxkvciy different from the 
testing of tlie tea coiqrfe{rf deciles. In the 1990s. mtHeediKrMional tests are 
gofaig tobe perfcrmaace-bascd and will be more fikely to measure highcr-onler 
w^goitiveAills than their predecessors. MiiIt^ri^<Aoice testing is nja likely to 
be discraiiniied in American or Canadian schools (no' shooW it be), b« 
format is likely to be balanced in school, st^ or provincial, and nmional lesUng 
programs by more direct measures of assessment such as writings tasks, 
perfiMmancc tests, projects, and portfolios. 

Our iMedtcti(His will hadly craie as s surtKisc to tcadters and administia- 
tors. The educatkmal journals have been fiUed recoitly with debates about the 
merits of current testing prog.-Bms; many school districts and provinciaVslate 
departments of educatkm have been reshaping their testing programs (e.g. 
Waiemia^etal., 1989;Roebcr&Dutcher. 1989); and the major test pubUsbers] 
themselves, appear to be following ihe trends by including more advanced skills 
in their standardized achievement tests. Import U.S. national testing programs, 
such as the National Assessment of Educational Progress (NAEP)., anticipate 
nKwe use of (^-ended item fwmals and the as^sssment of higher-order skills 
in the 19^ to fall in with wsw curriculum specifications (Qjllis & 
Rombeig, 1991; National Council of Teachers of Mathematics, 1989). The 
directions for testing in the 199(fe seem clear. 

Where is the impeuis for change in testing practices coming from? Undoubt- 
edly, the impetus for changes in testing as weD as in school organization, 
curricula, teacher training, and so forth, is coming from the widely held view that 
schools arc not doing the job they should be doing. Consider some of the recent 
headlines from U.S. newspapers: "Educators say public schools need overhaul, 
not rcfcffm." "U.S. yoydi fail maih test," and "American schools popctuate' 
failure." Or consider the findings in a recent international study of eight grade 
mathemaUcs and science skills where American students placed last OaDointe 
Mead, & Phillips, 1989). v . 

The sot national educations goals prepared by President Bush and the 
governors in 1989 are a response to the problems that have been idenUficd in the 
United Slates and are intended to improve education substantially by the year 
2000. The goals are (a) to prepare preschoolers for learning by improving their 
health care and nutrition, (b) to increase the high school graduate rate, (c) to 
make Americans the best in the worid in science and mathemaUcs, (d) to reduce 
the adult rate of illiteracy to zero (from the current level or 13%), (c) to make 
every school free of drugs and violence, and (f) to require students in grades 4. 
8. and 12 to demonstrate competency of higher-level cognitive skills in history.' 
malh, science, geography, and English. These goals are ambitious, and. to 
achieve them (especially goals£ andD. one of the main recommended acUvities 



tsto assess ^iideiitsfimby tf^^ifiiy 19 m^^^ 
peascmk^ ami pnoMem-solvtiig skilk As a residt, educaticm te^Lo^ in the 
United St^ wiU ne^ Id dm^ ctmngs^ what is imaramd (to r^kct 
c^BT^isn dmnges) ^ ttse an e}i|3^u^ 

the vaUdky of edkiraticmla^^sii^ Similar changes can be se^ in tte 
testing pnc^TGms in many Cai^lmn {»ncn^^ 

Ther^eiaint^ingb^b6CTl^)enedditfhestticasses^^ Tboughwe 
dislike the temi itsdf becmr^ we hdteve it cteiigrales ^ yeffi:s of impcHfant 
advances in psyctK»netiter^an[:h0^fcre^^ Hambteicxi 
& Zaal. 1991). many of the ^'^u underiying tti» new testkig nrov^rat appar 
to be soiukI, sikI the t&sultB will be bett^ ecka^cH^ a^essn^ts {x^icy 
makm, school mbninistiatms, teachars, diuteits, aiKi tt^jwents. Ob^tive 
tesiingt ^ rq[He^ced by tte multiple dti^ and tn^&lse foimats can cairy 
a stgnifkmit pan of tte a«^sessm^t (see« fw exmiple. Fair, Pritch^» & 
SmitfenJ^X but certainly not all erf* it As mcHe sod nHHeu^ is being made 
of test lesnlts in educational ^x^ntability (e,g., evalutfiqg tem:!^, schools, 
programs, and even states - see, f(x exam{rie, the 19^ NAEP Trial StBt& 
Aissessment at Gmde Eight), those af fee ted by the r^ts wi U warn 
the tests themselves aie fair and accurately measuce what stud^ts are learning. 
Even the most ardent suf^n^ of standardized achievement tests would not 
claim that these tests measure morp than a fnicti(^ of what schools expect 
stuctents to learn or It^ the tests can measure, with tl^ fiauttiple-choice format, 
all of the important higher-ord«r cognitive ouft:<m^ 

If teiK^hors are going to te^ to the ^lis covered by tests (and there is 
substantial evidence th^ they do already), cunent thinking is that tests should 
measure what is really important in a currkulum (Madaus, 1988). Then, 
teaching to the test is constructive and desirable. Shepard (1989) has even 
argued that assessments should appDximatc the learning tasks of interest, so 
that, whoi students practice for the test, some useful learning takes pl^e. The 
same argument was made fOT criterion-referenced tests in die 1970s and 1980s. 
Load up tests with the sktils that sttulents are expected to learn. Teaching to the 
test, then, is equivalent to te^hing the curriculum. What distinguishes the 
current movement for authentic testing from the eariier one for criterion- 
referenced testing are two features: ( 1 ) curriculum specialists are arguing for the 
teaching and assessment of integrative skills, e.g., whole language, that are 
typically broad and higher-order, rather than the narrower and lower-level 
discrete skills (i.e., behavioral objectives) that have been poputiu* in the last 20 
years, and (2) there is more emphasis in authentic testing on direct measures of 
the skills of interest Criterion-referenced testing and authentic testing are not 
at odds. In fact, authentic testing is simply a "fresh face" on criterion-referenced 
testing which highlights the need for the assessment of higher-order cognitive 
skills with more direct measures of assessment. 

The remainder of this paper is divided into sections: in the first, the term 
authentic testing is designed and the case for this type of assessment is 
considered. In the second part, the testing idcills teachers and administrators will 
need to implement authentic testing successfully are considered. 



Ito te^i^tio^tittlta^d^tfQBaf Milled fB^ti8saQibe» 
ifm9d.cip%iii««^i$vocei8& Mm^^m^fmmi^nm 

^1 ihft ftB^piteftiflit /^dtatejg^* 

8&8^ to itttss wiliiai^ and oral pmsefliato to asses apralE^ sScai^ 
wneaevw possiPw. 

lait<idtagaMr>amwtt,atedataaiwi^ 

paa««ea (pednpa aeveal ia^> ixte to ravoib^ tnc^iioiia. and. madi- 
oaptiea icsfa. stndo^ wKid lie ai{>ectat to iraric wi^ psotmctois, 
t ca lCTfti loi t t afltf aoinri^^to^ 

arawer* (!^aMei^^a^pftaito^M4itotfs<e^a8()<^iW^ 
ld^).(c)ie^[tlialiv«ddeoBttaic^^ 

l!io&«tfe«iiiMlii0fa^i^to«^i«itiefi^ Ii»fi»%ft^aeeiiti8t 

featom li iKit tiflkiae, dtlKT, 
initbeittic te^igadMKi^a^i^ toaeeapeat 
aaaessBfiBtta tat been tftonoim (aeaUi^ t99i{ Uiii*Baker» & Dtt^, ia 
ftm% l%reMn^teBfftM€k4isnfeta»attfieloii^iradel^^ 



a a s ea a wena . f^onian aid iflfonKtfoa oiiaBfvti^ 
ittfident iKffofnsram rad pio(^^ 
fe(»»(lii,atolastafew^tteiitfiffaawssm^iii^^ 

kisfiu^iOBaiadi^^ti^ttaftfflEeMghl^stoi^isnfta^iow^ 

aie jmn^ilBd. &)iiie caa be maife to mtift^^Khoice liy 




Imttfiaeatofiiaciiadtfiiii&astowtatcmile^x^ Ttei^ofiimHiifi&^ 



choice fofmias tokb i»m pnHnbe for ass^^ l^be^^ 
Attf^otte isi^ legate stoiteiis to 
exfieiiffleiit* paiti^ato ia a debato, aad so 

Tbe finoth IsaEBto k tte ino^ imAleaiatie b^^^ 



So^ft&atoie^beattiM^f^tUa ^^room tostingpmei^ 
am fi^ toSiied in pecfonnann testbtg fiiet^ 
acyeve In scfaocrf, aotf ceitabify te pioviocial/^ and sattoiud tesfing pro- 
grams. l%efBaa{b{fiQrofmotopeifofiinfiM4»sedtostiogiftft6pfO¥ta^ 
imd fi^kirut kvd lb a hoUy d^»tes topic at fbe mcmiei^ 
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Tte topic oTwii^tD MiiHKte leitb«lias liecdBie a fi^ 

wA the Anw&w MiadM oTl^i^ieit CAFI% Joteetf wtth ite N^oat 
Oomcfi all Mnsaem teBito6«M (lOAl^ to {tpodnpc tte AFT. fOlE, 

>. » fiof tea^ieCT comteethig a self-evahatkm irf fliek «tocatioBai 

^ a gn^ for &e de^ of testli^ wQdofaa^ for twites. 
4a^ig6ttvgtoqtecaktiiali^asi^^ 
biflaiCTtfie^c«aa^ti(M<rfgtiidejtfflatts8wCTia^c^^ 

lite Sfieifiird^ am iSQ« fKHMdoi to 
gdiMiPS <ag^ ftiiAimi & ftarideton, 1990^ because Acy ti» foU 
bacfciogofttotnoteatosaioas. i^so, visdSce sotneof the other elTom, these 

a&K tf^isplaf sei^m^itf fai^n^thm, <^ acllvitte 11^81^ 

fiiemtos^)9iasdscho(^^s&tetdecMc^HiialE^,utf<e)a^v^ 

wl^atB8{Vsinvohraiieatiaawiilw€ommta^ofed«»^ 

^S^A into seven bfoad aim 111^ each sea feiM desQ^ 

I.TtpachOT^iOtdttl»sldM^hi<^iiio>fa^«»BSS8i«awetho^ 
fof ktstntctiotial dcclskins. 




lesdtsoffxtfteMafi^pfodtteeitaRdtos^ii^^ 

4. Teaeto ^iM t» stdfied hi isbig asKfsaem lesyfts wh«i roddog 
deci^oiBi^boiiti^hiadstB{leats,^aii^t»chh^8Bdi^^ 

lum and school impjovemeBt 

5. t)Bach»s^caildbesldltedhiifeveh}|)higvah^|)t^lgfld^ 
which dse pupil asseiiiiiiieuls. 

6. T^d^ ^oidd be &iSed la eoMmtittleatlflg anessai^ results to 
students, parents, other lay audiences, and other educators. 

7. Teachers^ioiiIdbe^HedfnrBeQgitizh!^tinethteaI,iBcga{,aitdotherwi8e 
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AM Bit niem lt daBoed as ib$ procea laf dbtiMi^ i^aroi^on feat is med 

n^fiNDCdd BO liritf^ itf amen te^ fiiaaiee^ and no ^le^ 

(^erad. Btt«eeaiei(irafiiQad6r8»ofitemfinB^ka^ 
«lfaij«edfatf ofihecoo^^cnc^ 
QfCQiiiaMi|tdi|gfi»|{ieccn9^^ 

CIMdy fhea, mftslMtU ch^^ in Ji« piMcnrice fc^b^ of fiacbett lUe 
cffiiediQrtoiBeet]^ii^dieaie^(tftt^Bew&^ 

Shttnnttty 

l%c» fa a aeed to edaiaftia feAK^^ 
i^otttat tbtt <aM &e ^asaied tsKffiiimal filial 



(^ffl^ P^a^eliicatogiwiiddfect^ 

wneieasttoi&assKssmaitamnia^glhes tlwfartestfi%{^ theato 
locate 8^H«deiAiteeaiidies©wiMtodM#de¥eI^praw^ 

aDd.^«)tfoaiteai^rie^aittesLji«soemes(ari^ 
s»8«gtolasB»toea»»itidMsBeBdDnv!«R. F^^3 is booed di« the 
i^g am«»iBicles KWJiiineiHfed ^ 

asuShmm testing movement 
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SMw^ Ri^iQi»l BcfocatiQB^ Lto 



these vaikm adiite, I ^ sevoai kern to 

1. the Bi^lefi^siKb the leal^ of cbissn^ 
hh^ iffl the ^latoy «rf the ieK*^« own chfBiww^ 

101 



A the lEtiaer wdenttoli ^ taMMpaiM oravtfWiie ase^ 
nirat raetfnds mdfcm 1^1^ dioftt i^^^ 
Site tn^eKciacom|»»^tel«^ 

& ttB^ffl^ ccnrc^ lo tewINn fiKl tfM 
ment en raake tbckjob finiR^i enfer t0^ better* teflutQcfe, 

sni iBscivicemfa^ctnieidi. 

tetiae«99mdmea6ti^ik^.ii^lUblg i^eettki^ fa^ 
these ^a»iB<fs aMi ft b oftlnl iri ds la 

at ifie iraie 1^ 0119 t«i^ 19 llirw fntmi^ ifl^ 
nHMtti^RMimllheimefpaiciiiitf 

ii}oinf.teoid»«iHMiB«(tein<l^^«ifi(^^ 
6»ei8ii^tftatlMd^iieedfBk^. Btttniofi0iii9oftaRiIy,ifBfasfswtocaii 

wtfBiigBctwwBaw ■wciimenl «n>iB«t«tgtecflc^wdy. Without that credflJ^^ 
f ty« ite iiain»*i i»»nage siniply wHI mtl fftfiNtg^ 

liefaigof i^idHits, an{ sb^hhI fatvolirfi^ I^Kflxtff and t^r sa|i»i^afs, 
wte ouliiBte tlw {Kafonnfflce of (oefKm 

With le^eet u> iOuta wd!f tetog. laichas its^^ 
ifiey sm ta fo^ at the cent^'ctf fte asiesimem pi««n that fe^ 
ihe (foclstoni tfi^ ei^ the giealest (nflim» of leamhig md arad^ih: geff 
eoiaxft. To fBiutntte. teachen thonsetm i»e the t^fs of (by to (by 
assessments to diagnose student needi, group students, gmde isttdettfs and 



ttdffitlQO^ fb^ tCK&QI^S ChSIBSWBi WSffSffiWltli ^rovjUc ftO fl^SltSMltiOO 
ffjwfr*^ Bfft^ t(y ffl^fcfr flWIift CiWCill df¥faMflff fiWf Midi riwrtit tflPltMBlVCTjif ^odi is 

liiftttacte^/teai^i ^p BPC g ^ 

$bostiL Aft irffte» tfeciitffMS 1^ flit flie fflgfliiy of rta^ teirifefV ^ly lo^hy 

^^^ ^^^^ iWW^r fft^l ^ff?BiMf C^Ufft^**^ ^*^ *^^iffftt'^4ff 

time: nsfvsmm. Briad^idsaiefesinBsS^te 

tlie «n^i-N3Bg i^stttf^ littBft Oil ^ ({11^ t^iea^^ 
^i^CTt nj&wQiKBt t^cadMsa aie Bot^ira^ l^dwd 10 n^danzB Aat 
^lal^. Ttey miMcaie aboot tills and iialEBqaaUtya|gfaHityif<|ti^y is to be 

Fof decades now, fee mtaimremeat comannrity has ffictalai the figiBgt of 
assessment in schools. Tbefia(fi&iiialdi^CTitKibc^*TUs&fb68Ssrasnieflt 

thbflietfaod.'' VtecimsAffhtPite;m,9^amomhm6CO^totes^i^ 

t»um^a^hm^*'sceeili^ianm. Nofvlht^faavebe^tosayb^to 
the sneasur^^t cfflusfianhyt "Ttese are our achievemeiu taigels — fSbeyase 
fi^aiiddlveE»a«idoiioAfitfiiefnkfii^^ Give us methods that 

fitmtftaii^** lb be ^llBdive ia dl^li^ a^es^i^ fic^i^ (o t''«^^ 

meitods. 

substantive sob^ inattBr kztowied^ denK»sbate that titey cai use that 
knowled^ and tbdr h^ber order thinking sldils to solve problems, exhibit 
coiahi Idrals (tf acbieveiiKni-ffG^{ed behivt(»s, cieafe achlevefflcot-rcbtted 



frodnctt ttit posam eatsh ipe^eit mtdiwiw, ina i^nSaeenda rffectfcje 
jqod acieoce» goodi fl^^ gt^ ttP qm^Mtod nnwwitf wtoCT i^Bpit 

iC>Cl?f f pitjpM'Btt0lli MOWB Vf f i it iWlfWll^P Iff ff^llfPft WfftllHfirt l!|W!H!i|llfflf 

flSMMm^ of iteiieftt acMeveaest to the ^mtom ttn^^Hm ito 

wMiraiievriopHilvtfi^leate. In^t^adiefsteveAisdfydineiixif 
iBge«mefi|aplioii8tfSlidrdbpo«8lfivatte$sbtgfltu^ 

UmbatfoDt oichk tfie fcej^ ID ^ effective de^^ 

piifaHs to jQUndggiMMneflt III carcase. AsjusiU9ttlni^^{sofiidBR^teviuit« 

tihm the ttitoer esfi eommimkitfe dKW dfSiDitt lo teadias cISN^hf^ to 

l/inflwitt lai6wledge of 8& sraSd^ s^cssi^ 
Onfl^rtn^ on Qw^^ 

tosfexitrad tiie ddEMtfims Of varies (tf vi^^ ^ 
lraiwliimtt)iiiBiUmizBd»^ll(yitfiteirfisse^^[UsCU.c^^ 
of invi^lty mtd amidNiffi^X ^ (c) ^biow how to ^ikaate statistical 
«aiii»lffiSi ftf qiaUty, » as (0 lie ^ to asse^o«i& ^ fiK^k^ 

accq^ t^dai^ Th^ l^^fc^ represent the ide^ when & comes to 
sddffQEifajgfftHittjrixNiliuIfesiiKiB^sessiitett. 

B^tfcofthedrfbiitiMgairfsiaifaticalesttaBtffiHitirocc^uesti^^ 

and d^cattiwii^ flawed BflMMiittitftiiiiBayaMfefipf assess^ 

irimpiy b not neafistlc to fiold leacbns am»ifltBUe finr meeiii^ ^ of diese 

guidelines. Assessment training b made relevant for tcacheis when tfa0 tr^s^ 
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InowAbofisoincesofransncBieotcnori^ 

iss^leciivcL CMwoi^ ia&»saii()4 ^ modMs tfae of Aeir 
tttcMBigm, tte)riloiiotiieedii»km<i^^oBScf«stoii»^^^ 
aiiiiiM^(aitiyaalm)ivfaowtoc«iaMe 

Oetteiewirfteteeiiiicd SBppntpd waled toM^ 

l&8siiistf»ea«ioabQimcafiy<tH^^ 

Aod Boiled daasniara aaat^^ 

d&aob^ aeed » toow flj^foteolja ^fidtoiwd ftow towt^ iftan to v<ay 

a sses s roeitf taiiierai»if»»oaltosceii e de ma Hift ^ tfim 

anay of acti^^ i^ted tobefiff a^essmem to ftteMy fidl agendas 

di^wiB«filbeHs^iedt08«l«tt«oe»Uieyfii^8id&reiii^^ 

fi«n ovcf-woitpd edttcainrs. &tib^Gc»last,&6cliaUra^isio^sa»e«i«9 

io^lii^h0wieactof3)eiidtfae&po&s^i»^Aii6. Oaam^myta 

doAisklopEOfi^teadmttatf^s^BatbdbssRxxni^Bsai^ 

it pos^ fir titoa to do Adr job filter, caster and in fhtffi^ 

^to«riym«stwggNwdlCTl^tata88e«s^»rtfaaotathl»ca^ 
ccavia^iheiHthatcaitfHlasscssoiemcanbeaniiiro^^ 
Thb ctn 0^ be aocooi^M by devdaii^ a o^ecfiQit fli ^ an^ 
sai^ ideas tsafaaieditfb^tc^ili^ of wfiic&^(^ Iwybiiciear 
laigetsiiwfces assessment C88ter«nsfaigl^esfrffe8tspecl^^ 
aadpea^fest^ens&ii^o&itaM^ efeffirfN^fijiaiiaiioecEteriaimlasscaBiaioid" 
cattoa msSm sid pacems lister aod easfer» i^i^i pntctices 

facticsafedieiiigldi^oftralaiBg,fBsessai^tfi^^ 

Assessment training aba is fmie most retevaat when tite trainer t^es 
adfanaserfii^niefeMalsfial^iestfwttetgBgsteato 
tiiatmc)dedassessmeDtnMlhods}npim:ti(3l{erD^ Tlwse 
can involve assessmoit simuMtm, evaluations of previousiy-developed m- 
aessBotts, appltcalions of the {ninc^rfes (tf sotatd Kses^ient lo leid-world 
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imU an^pd AciD-^ 

tiabef fflosi diomea^ ei^^lii^ Mf gnA> 69 M^iewuntst of Ktcben or 
^elr dwg^ dnmlag fotowes fior teacbas » cnry M to tte^ owa 

gcealiie^ of om-«t^m sid«n^, wcBHislsto &ce ^ vaa 

tli^ofleretftfiemf»asses»iaillcri^^ SsiratlaBy.iiwhaveracatiteji^toeal 
lieitl^in need <tf ii^iBVMtt, hp^rf flssegaHCTt tedUlng. Om liinited ftshdi^ 

latoi&SL-, ■■ 

1!^ letees 10 dieis^of fi^df^ leM^ ve^fiM for tes^iefs Wcffi»e it 
foice8tiieteEiiii»flKftiMotl»fiBdliiwi(land£^ 

Esia^boiir ^BdilSiQr fls a flDVcei^ia^ ideas by {HCKfiliig 
t6^ ito teaeb^s ofiea fittle fAiiyf, «e veqr aaiioitf 

in the sense that tlwy actuaity do change mx^m* mg^mmm. vaima awl 
psacttoca. TminiBg develi^^ed to ivt»i in thia context can woifc anywheie. So 
tlfegqges^^eaicellewptece 10 start. 

€kmc{\akn 

We live ia a sodely diat im becoBie frfi^sed tite attmnmeot of 
ednKSlifiMid mfteOflosSt EvfaS^s of tl^ obsession abounds. N^md ttid 
iitieniatioaalassessiiieat8frf'oiitcoia»tfi^w«eifntieaid^^ 

Stsia«^ asi»«n«»tts bai« grown itt lamibef over tte 
i^ariy every stale and to chdni total costs tteitfsrexoeed the c^sls of the iiali(Hi£d 
ana in«nimfHiffl asKiessitteBis. i ne ot saattaaiusseu aciiievenicut t«tt»tes 
in local district wide assefflment pn^ams contimas to grow to reccffd tevek 
In this kind «tf enviiuiiutent, b is both {^radoxicai and veiy tnniblnig to 
iGali2» that we abo aie a soci^ fltat is ^ffic^ oiini^letely filte»a^ 1^ 



S^Bf>^8^.&C«ifclto,NAta. iimf. Teacher TfiA»&«faAiwiowtt.Ai» 
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QUO Boc»4A IS Heads above 
PUBLIC »::hool counterparts 

<pte»il»^iiU 17, 1990} 

TEST SCdHIES MiSLEAE»NG 
PRINCIPALS SAY 

SCHOOL RANKINGS IRK EDUCATORS 

P^H^ «|aci^ ^ not s^ved &y radcii^ 
tfial stB$^^ achieve oo i^itotisit exams, sa EdnJootoo s^ocd 

1MlB^*si£»n^(iitf indeed M^8cfaoQ!soa^6^ 
exam lesiAs. CBnz^ 18, 1^C9 

At y«ar*s €mfei«ice flB Ctassnwm 1^6^^ 
Vidoi^ I osicttidBd my with tte 

0»l^ ^ stiOiS si^^rat for t^l^ ind assessor 
tite le!^ <»mffls^ is poviaee as ^Kwteie <s^ Iklsi^ 

&m. fteexffi^le,westwtwD<tfdieadvi8Qfyieseaii^tesTO 

titsiit. Tlibf9tiai^coiitta^kmiso^eiedmdiewfafH|s|e^i^^ 

feadi»s(eadiody wfaatisctmskleiedSk^lobeirats^ iRtfie8affieveiii,ies& 
^ fineqoei^ ai&^bcd &r f^ias^^ fi}rai^ig OR flid iHri^ 
test: ie(^c^&^!sdMaima:&^^p|)&»th»<€«rotbiea%^ 
do«BS,ffiMittettE& WeaeedioteaN^iil'»K^o^«im,We{i^beiiitfated 
to gi%« tbe (!(»feMn^ die pPMe(^k»i Aey desert 
^sesiESfflsat, ew^MM ^ 9«u^deed afid fo firassat tt» fiAftn^ 
mieitm evi^ die fi»x» ^ li^ f»e c^«»s»ed 
numlMss. Qbogm,i990,pjSa) 

Tbe £aciis cf tito puram fiqief is apm dds f^i for oramiimei' pnsa:tkffl. 
Itifore sjKidfksBy, seed to te tdcea to jmi^ 

poisiidaf mls^ frf" ies^flg and 1^ ns»^ 
tie£ms address^ ^ cpiBsttioii, & 



TEACHERS, PRlHCiPALS Cf^iTiCiZE 
DECISIONS TO MAKE SCORES PUBLIC. 

(Fbrnnit April 23« 1990} 



Ammasr indeed beca»0&iritoiB. On Apifi i7, imif^S&mi^nJ^mt 
f»d^ i the fiN^ Maided mi^ deft^ « the iv»age <^Ao 1^ AOeita 

ltein»iiBfi^ ^wvmsslt^ma^^ ^ogU^ 30^ 33, imd Social 

by ^ co r respoiid fa^ csds O'teu^. >^ 17, 1^ Six da^ to the 
famsi p^rttM si^iod-b^scta^ sisMfts of tfie A^&m provincial 



afid9^^0BV&i^ciB»^mspeii^ffiMinisiberof^Kbnf8 who wrote 
the exsminafiQo wefi» Msied £^es(A jmhtte a^ofrf BtooatOB stii^^ 
ittlhe(»ii8qia«St^gr^lev^ Ualifcisthedti^i^Su-l^ajha^scNx^ 
ia^weteitotK|wited<nMmiA]}r^23.l99(^ Other than the foUowii% 



«^ fi^xsted, 80 ofh»' M^»ma[^ or d»a were i^mted 



tests. Acn»v8^ittesf88set»»f»Bdttipi&eith»fe&c^rar^iid^ They 
sue t^ ai B nKasme of fi^e^ ^ ranrfoohan tas he{» ts^ ami Iettii«l in 
tfteaimtefoed. IfUiBiH»Qb^of8tirile«s«in{mithese8,istes8th{itt25,th> 
i«sid»^niIdbeiirtefi»^whhc^ioa. CBi^ertiA|ffil23«l^. 
AsptMi8hed,ihknpoifii^(^tl»$cfaoi^M)y-achooti^^ 



Psyek^gkat Te^ (Amer^ Pliychoiogfeal AKOoiadon* 1^ ami in the 



tion, I988),paff of m/UA is stown hi Table 1. Wtat ^ theevid^ tosiippoit 
h^ clfliin? 

1. the stalasm by Albma EdtM^tion, the test developef, that the 
r^lterfitsPwvtadrf Diploma Testing Pft^am^ 

aie ml hitended to be is^ K> nma comfBkteom anmg sdnids 
(Albra^ Edia»tion, 19^ p. 107; 19«lb, p J), the test iser employed the test 



fichuds peifQfflied coiMdstently betow ptoWiKJlal averaps, the central school 
admhiistfatioR of the Edmonton Ptd)Ue School syittem annotmced hi early 
Jffiioary, 1990 that diey would release provincial ntmninethHi seoi^ on a 
schoot-by-school basis ki an attempt to reverse thb trend. Prior to this thne 
individtial schools 1^ provided to theh- parents the results (Stained by the 
school. 
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Stmrfirfe for iMtwtilflmI tad fwclwlMhilTtfbig 

wiBtxi tiled. (IVimi»y)0^84) 



HntHitioitf of fqml^ Mm* 

1^ my ttotini or compirtaw 

ttio |>m6U tfscd to idoGi tho usiptei 



Tauten GRtosbb 

9. C%t^ tefbnmtbn ifxml the teak mcd for 
icpofll^ foofo^ Ao 4»fiN3tori«tei <rf «V 



tfi^^ dMficiCfiosi bttwscti tfis 11(^1119 or 
tik(^9» Alio udfo ioio Aoommt mjf 

thfitett. 



deyelc^mleti evldem^is ofetaitted to 
«uppottlhe intend^ oi«u 



IZ Proviite faiforaiitkm tfut wiQ he^ tt«fi 
£i^ow fViioittble {ffooe»h»«i £mr 
piiiif^ whofi & If qspnqsfUfe to 
to ttio fitfiii fcms wkh dw toit 



mdgitfwr evidence to ft^l^cit the 

i^^pFOptetfic^i of ths Hsot&^ 



tS« PwidebifofmtkinAAwiQhe^Ofeff 
gather evtd^m to rfimr thil dio tefl ii 
smtteg iti iota^ pitipoie(i). 



13.0t^tin evidence to he^ show tittt the ten 
if tmseting 6s intomted {wipose(f ). 
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fiomtt 




pai^^ is to idl tow iwa Of |»oriy 

^^ww mario Mrf a^Owwiii^t teit awHw to 

coa^aadv^coa^a,^. Sdwotoiglswjwiteiitheiolaiivdyiianow 
^itfIe«^«wicoiBfisa»eM8dbyitetMi»ssffl^iwt AMw^rfaese 
cwteomw ^oiAl be te^ and teaiaed. scharfs are 
o^siaMS th^ iomd temd ttosa asse^ by tsmm pmvimM 
im^MtaAtteita. Siairfy lat, m tte oriy «hw» of hrfimnadon amm 
M^attoi»i^orty|Mrt<tffte Story; fdUagtaliiOTiwfa^ 



fecftw jKAafi^ varfois demc^i^ and opj»itQnity^to-to} variables) 

been slwwB ia Offer aiafles itfs(*c»l efifectlw^ 
s^oolpgfonnai^ '^wseJi^ia^ikmarefflrttocoinplfc^wheaschooI 
^ffflto we bs*ig inada Aie to hwvftj&te diffe«^ 

canpatas Bfflde Is ratetag fioin U» 

i;. uit^abseoce^oseiiiinc^cominffiitod: 

^To a^atdy compaie sdwob, you need a Ifflge an^ 
au iftal . (C. Uaal, peraoi^ wmimOTicatei, 

^ISr ?? ^ ^ reportfag » (^herwise tenoriiw the 
wiWWig^WienDtttaihectea. If fa«o «x«jeiit, as tova in for 
30 the ooaveiskm of mato to die school rai^ 

ot Uie Gulibiiity inhemit in test scorns. 
mstri«i>eisoimel,ediaaiton^ 

"!r-S"^ ®^ ^^"^ tafofmatton if discussions about education 
aiHlsj^ingafetoprodw^posUivecban^inKJli^ MomiaikMi tiiat is 
flKomjjlete m form and content win be counteipioduciive to any educaUonal 



STANDARD 3^1 • Wto tesNaSdim 




luirelated to tho coostriK^ or ccstcDt n^^asom 








eiqpI^Kdl to te^ tiloeis be&»e left sAni 


nisicatdelihtsrinan 




m be made Ini^, 


along with the ^ diuBCtions. H^useofsocb 








perfocmance. (fV/- 




J98S,fqi. 27-28) 



niustration 2. 



toimpt)veamscaeBOfl»^wtett^coM6filarai(^ale^0S^B»ad(l, 1979; 
Besson, 19^. Ifat^te^pOfis^Ksl^'iN^iaMSii^tf iheexamfauolGn 
<KBrt^ stts^^jbie ites^, tt^ die omibtotion t«w ^ raniU 
in ffii fanjaxrved £»»e; in (»»ura^ a ^^^it t(w to tesl-w^ness wUt 
p^iafixed6¥sxytbnel»a'fil«t^»sateft«^lehtoclwtot6^ 
Tbtis, a p(^attia) vaikfity i^oUem exists wbes (me aa»ii|}ts to faK»i»et the 
meattiflgof fhetest^die. Ifteascof^caittwiAfitteiicedbysRiid^b^inme 
or less te«-wi»5, then those individi»ls tovdved with test develt^nnent, 
administration, and iuteiprDteticm need to carefiilly consider the construct of 
test-wisen«s mid how it affects scores, 

A number of authors have offered definitions of test-wiscness (Diamond & 
Evans, 1972; Ebel & Damrio, 1960; Gibh, 1964; Stanley, 1971; ThomdSce. 
1 951 ; Vernon, 1962), hot for the purposes of this study, the deHnition proposed 
by MiIIniaii«Bidiop and Ebel (1965) has be«i adopted: "a subject's cap^ky to 
utilize charffljterislics and formaL<» of the test and/or test takkg situation to 
recdve a high score" (p. 707), Millman ct al. further noted that test-wiseness is 
Ic^cally independent of an examinee's knowledge of the subject matter 
measured by the test's items. Basically then, test-wiscness encom trasses ootn 
the method of measurement (flawed test items which p:ovide test-wise cues) 
and characteristics of the test taker (a cognitive ability or set of abilities that an 
examined might employ in any testing situation regardless of the content 
measured). 

Millman ct al. (1965) included with (heir definition a taxonomy of test- 
wiscness principles which im served as the gei^ framework for further 
studies of test-wiseness. Briefly, this taxonomy is oipniz^ Ltto two catego- 
ries. Part I contains elements applicable in most testing situations and which are 
independent of the test maker or test purpose, Ifempk^ed^U^^cat^i^wiU 
help examinees avoid losing points for reasons other than lack of knowledge of 
the content tested. The principles listed in hut II of the taxonomy may prove 
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benefice wli6attetesttteiiMlmi4Bril^of|]i^^ 

te^siroi2arfaiptti|x»eandf(Bsat 

Attte»gl) Uiii^toeiX|}^0diliAst«^u«aad tests dev^^ 
^ffii^wonkltefMv^ iffiiniiaeiotest-wisea^iieseaR^coiidii^ 
lMedSfat688i^liBst$diattlibi8ii(»tt6cise(Bffiigat4^^ 
l^;Bai«ii« !988;5an»cid, l979;$taGk&I%Hter. !9^,&iiM},19^ These 

Asc^tos lint oo tirade IsKKd (» j^cHw ihm exffiiyiiati«m 



Tables 

Stnamsy of Te^Wisratess Analysis 



ProvinciiU 


A 




Mean 


SO 




i 


Mean 


SD 




BqglUiiS 


73» 


ii 


$7.9% 


i3j« 


J» 


7 




19.7% 


.20 




»? 


25 


61511 


ISM 


.71 


33 


4f.t« 




.73 


Onmpfiy 12 


m 


41 


I9.5« 


IiJ« 


.ei 


16 


43.t« 


UM 


.48 




m 


39 




I&7ft 


M 


to 


S8.2« 


20,1% 


.46 


Bidogy 12 


261 


17 


64,6% 


15^ 


.SO 


15 


sa8« 


m% 


Sf 




174 


28 




13,4% 


.76 




S9A% 


f7.6« 


£9 



md ihe lE^jicfffontiBic^ m the i^m^^^ 
each »i;miiiua&m(cTO^ ^4), 
^IhtcaoQal coos^leRey (Hoyl» 1941). 



ffiS tfii^inuiiarv fBffiius « a stimv m the mniifiiK» nf t(^^ 
wisoi^ qxm sttMtett peffonnaRce (m (be Ch»de 12 piwtad 
ii(&riinista)BdlaBriti8ha]laml]ja(Rogers&B^em 199 CMy theiesutts 
fl^eakfiH-theinselves. Rnovkled to TsHeSM® foisrex^plesctf flaws! le^it^ 
itom the Algetai 12 examinafkm which cmitain te^-wl» cues. 
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Tables 

Ak^n (4-267 

Use ^ firftowing (yf^EBB 10 s^wn* 7. 

tt 



(2,4) 



7. Whtehof the following «Hildbe the^p^kmitf the ftnwtioR who^gtf^ 



A. 
B. 
C. 



3.7 -.19 24 JO Absurd option 



16.1 -.30 26.3S 



10.1 -.21 26.70 



70.0 .45 34.27 
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22. W^t^0^k^btgiammitk»^s6ipmK^ 



B. 
C 



642.24 



5j$ 

m 
1.1 



.26 
.01 



23 JO 
3254 
26.63 
32^ 



24. 



-8.4.-2, 



sequence 





» 




A. 


4 


00 


B. 


-2 


1X7 


C. 


-1/2 


86.9 


D. 


12 


04 



aoo 

-.03 
09 

-.07 



OiX) 

25.^4 0|}|»sitBoptloo 
32.76 Oppo^teoptkm; 
22.00 




37. WWch 



A. 
B. 
C. 
D. 



y=cotx 
y»csxx 
yasecx 



11.6 
5.2 
3.7 

79.4 



-.16 
-.16 
-.14 
.28 



illusuated by the 



28.29 Opposite option 

26.36 

26.60 

33.01 C^jpoiite option 
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idiom A^linKil^iiieastaiBMmh^Cima^ ffw^oMionfaecomeiAiienRnfe 

Bducukm If Mthtomom^auei^ttefsaixM^ttiaGdvadiiii^^ 
f8i{iiettfor|»9(»itolbrdie<tevQbipim^ 

^H«wyiiidnimimyfort3»i«li6-yetr-^ I!i6f8femiil»tlte 

ingfimyotf 8cieit(^a#ieerifli,ortol«oiosr. SoftcidGttmdtenfXOVfaiMS 
Witt l» paitte^itting in ti» second btem^faMtd Aoei^^ 
Fl<^«s^ eaitt» Ms namA ffovta^ wem ffi&ed to 
lliM teiemaitoifid Stiid|$r of Mi^i^cs itf ^ 

Wlist tiiose fesBte im«at to ihm ti» saxes scftieved «m ti» |m>v^ 
oxairiiiathiii cm sot te ^qiiy md vafUSy tafapet^ as a raeaaiie of tfie 
cafefn»tofte»^ b^e^0»^Hesofnfny<rfthe^tosaieoofi£oiiBded 

AMmmi (ISA). Hitottstb^ don m mm^f^tea^ coiidiicted 
by schofd cottiiselton, sidH)ol |^wlio!o^p« tfMl 
dfo&ii^,86l»Krf&Mrislfi^s^«idf^swoomtracto(W^oii, In light 



of fti»e (»pnMoits p^l^ (depfiid^) sml vaikS 8coi«^ 

and seem tf^pfoteUoRS? Idtbdcnot. 

Tl» ^ <^ dite |»|)^ to tfM a mmittte ci»^ftod of f^K^^eBi^ives 



prli^fldes gcmeraliy accept by profifiakiriel mixklkm u tatScAive of 
soaad {jfScUra — f«^ tCKhlsg and evahntkm pnMto fai Osia^. Snch a 



usmm and t^tbig ^aiufanto wMch allow diott bivolved in eoRimMoning, 
eonductkig, e^ng, and dtosmh»idng educational evatuatloiM lo use diak 
judgnmm and creativity fcsponsibiy , but widiin dte botnukries of "fwcqitebte 
and sotBKl nt«istif«natt evaltatkn {musSiee.'* 

Membership. The ofi^izationsreiirBsemed on the coainiit^ called for 
^Hnild oichtde ^sociatknm whsm members amunlssion mid use ^ucaiiiHi 



te^ tmd le^ing jmigimns, and assoctotlons wh(^ inei^wis ffie affSected b^ 
whose woifctodtesnbjectd', teeing andovaluitifm Daeie^nnt^begiv^ 
to ho^ English and French rqpresQitetlon. In Caxs^, diewossoctodons might 
ifichide die Coimcil of Ministeis of Education, Cmndian Edoc^tonal Associa- 

o in; 
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Bttoc^ioii. OBRsSan Oirftim aad C^S^ffli^ Ami^^ 

dKdq^al Associatkm, Ctatiafl GoiBdl for S^m^ BifaK^n, the 

Canadiffli 'Hrachas* F«fciatioa, for example Q>fm 2), 

Concision 

lliecaaofiliispapBrfefwaiaimiittMi^^ 
as^^kiammdfaqiortama^toii^iiovii^iea^ l^nm 



of 8tffiid»iib ^cwfd be to a lof evahiatiiffi tke ffl^ 
!»«»k!es as Oey affect the vartapffithstevQli^ 

Does the group of indlvidoafii here support this calJ? 



Reference Noiet 

1 . AWiottgh a#i^ obltiiwl le« SMiies for the tea 
^q^mitiaAy-lo-teani iaem sad ifieii cospsteg st^oc^ is a fi«ffl»m}y 
used techsi^ gjr kt^ttifyhig ef&cave and Ic^ ef&oivo sc^iools, sich a 
^«(«toiB^roa8icdesfaedfB^nhigoit(s^^ I^irexttni^idBdemsfitmi 
tea afflMaf fianaies ^widdaiiteve at thesmhlgh te^ CM the achtevw^ 
6^|MsiBdsite(h)mimTOa£atwntlBiiiMesCaiw^ I WjGusltey & KJfcr. 
1990). 

^a«iBm^qqxrf^bylheflH«,^xin^^ 'HicAiMiteffli 

^atlmd ResBfflch AModatten. the AiBcita 

atid fte ^Itfiimal CSonsi^ (ffl M^uiifeinem in Bdt^^ 

TkeCe^efFt^TestiHgPraeUminEdMiU0hnwmd&V9k^tfyMm 
CteMiWa oa Tesitog ft«slces ^xms^ 

ResMfch Agsoc^itt, Americmi PsytM^feid Anociatton^ Notioiml Ctoandl 
M^o»oe« in Ecto^^, Amt^ AmsMm far Coanseiling and 

Ddv^^^m^ Mid Ae AiseriQin l^weeh-Lttigwge-IiMitg Assocbekm. 

pewistoiheUii^Sta^aJolmCimiiiitooBSiMi^ 
Brabtatlon. Est^diifMdin i975,lhisJfolmO»ffliteeefecra»iidy mk^^ 
ffitoMijo^iteaalediicatfcHialaMociatloiiilM 
the SiiKfy oSmmkm with the assonance itf tlis toiUy of Edited si the 
untversttyofAlb^lopmiiioteeom^farevatttatioRofbighqttaU The 



ml Bva&aittm Smndefftk, Vm Commte b mm be^mfaig to develop 
^i»^s for ^ evahaaiafl of stiKfffita. 



esamtnatlom {^grtmi. Edmonron, Albcmi Atfa^ BiNcctfan. 

Afljoti edtt£itkm< (t989b). Provtitelil RfjKHC, jtme 1989 Kfaointeririoni 
Ac^vMitcaittfiMtiiitnoiWRt. SdiRimtim« ABmi oi Aftt^Ethicitksn. 

Anwrtesn PkycHobtM Aiiactttkm. (1988). Cede ^ fdr mtb^ praoto (n 
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